2024-04-06T02:02:20.761 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-06T02:02:20.761 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi --verbose --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836 -- /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/orig.config.yaml 2024-04-06T02:02:20.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T02:02:20.879 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7641836' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 workunit: branch: squid sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5609 sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 sleep_before_teardown: 0 subset: 32/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOK1pZtk6p3UHPBrlGCf0FkELt6+n1I+nY2qQNy86caQnb3ObNn5cXSqgvMLCLhK03JKTYZz8/hZNl+sEDU26cI= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3Iwr0j9jUZp0aTpe7Fleo1rDVSdsbloWkXUPswxzGAwq/g1Sp7fiQARvCUKRc77sjbFiaH5F7i/JuiF2EQyRk= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-06T02:02:20.880 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa; will attempt to use it 2024-04-06T02:02:20.880 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks 2024-04-06T02:02:20.881 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T02:02:20.882 INFO:teuthology.task.internal:Checking packages... 2024-04-06T02:02:20.916 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6' 2024-04-06T02:02:20.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T02:02:20.916 INFO:teuthology.packaging:ref: None 2024-04-06T02:02:20.916 INFO:teuthology.packaging:tag: None 2024-04-06T02:02:20.916 INFO:teuthology.packaging:branch: squid 2024-04-06T02:02:20.916 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:02:20.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-06T02:02:21.174 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2027.g9f192350 2024-04-06T02:02:21.176 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T02:02:21.185 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T02:02:21.185 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T02:02:21.211 INFO:teuthology.task.internal:Saving configuration 2024-04-06T02:02:21.222 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T02:02:21.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T02:02:21.257 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 01:57:11.590167', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOK1pZtk6p3UHPBrlGCf0FkELt6+n1I+nY2qQNy86caQnb3ObNn5cXSqgvMLCLhK03JKTYZz8/hZNl+sEDU26cI='} 2024-04-06T02:02:21.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 01:57:11.589018', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3Iwr0j9jUZp0aTpe7Fleo1rDVSdsbloWkXUPswxzGAwq/g1Sp7fiQARvCUKRc77sjbFiaH5F7i/JuiF2EQyRk='} 2024-04-06T02:02:21.281 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T02:02:21.293 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-06T02:02:21.293 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-06T02:02:21.293 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T02:02:21.370 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc37ee280d0>, signals=[15]) 2024-04-06T02:02:21.371 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T02:02:21.380 INFO:teuthology.task.internal:Opening connections... 2024-04-06T02:02:21.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-04-06T02:02:21.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:02:21.490 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-04-06T02:02:21.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:02:21.578 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T02:02:21.587 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-04-06T02:02:21.605 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-04-06T02:02:21.606 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-04-06T02:02:21.663 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-04-06T02:02:21.664 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T02:02:21.664 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-04-06T02:02:21.664 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T02:02:21.664 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T02:02:21.664 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T02:02:21.664 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-04-06T02:02:21.689 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-04-06T02:02:21.708 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-04-06T02:02:21.709 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-04-06T02:02:21.765 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T02:02:21.766 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T02:02:21.766 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-04-06T02:02:21.789 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T02:02:21.802 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T02:02:21.815 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T02:02:21.815 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:02:21.818 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:02:21.833 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T02:02:21.847 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T02:02:21.847 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:02:21.875 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:02:21.974 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T02:02:22.026 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T02:02:22.046 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-06T02:02:22.046 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-06T02:02:22.046 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:02:22.047 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:02:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:02:22.063 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-06T02:02:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:02:22.064 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-04-06T02:02:22.120 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-361.el9.x86_64 2024-04-06T02:02:22.120 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-361.el9.x86_64 2024-04-06T02:02:22.120 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-04-06T02:02:22.122 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-361.el9.x86_64 2024-04-06T02:02:22.123 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-361.el9.x86_64 2024-04-06T02:02:22.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-04-06T02:02:24.404 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 12 MB/s | 8.1 MB 00:00 2024-04-06T02:02:24.525 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 11 MB/s | 8.1 MB 00:00 2024-04-06T02:02:26.355 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.5 MB 00:00 2024-04-06T02:02:26.450 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.5 MB 00:00 2024-04-06T02:02:27.999 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-04-06T02:02:28.770 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-06T02:02:29.966 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-06T02:02:34.669 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-04-06T02:02:36.210 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-06T02:02:37.295 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-04-06T02:02:39.746 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 21 MB 00:03 2024-04-06T02:02:41.587 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-04-06T02:02:43.071 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-04-06T02:02:44.751 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr:Modular dependency problems: 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 1: conflicting requests 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 2: conflicting requests 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 3: conflicting requests 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 4: conflicting requests 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 5: conflicting requests 2024-04-06T02:02:46.875 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 6: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 7: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 8: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 9: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 10: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 11: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 12: conflicting requests 2024-04-06T02:02:46.876 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 13: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 14: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 15: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 16: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 17: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 18: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 19: conflicting requests 2024-04-06T02:02:46.877 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 20: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 21: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 22: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 23: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 24: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 25: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 26: conflicting requests 2024-04-06T02:02:46.878 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 27: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 28: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 29: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 30: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 31: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 32: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 33: conflicting requests 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.879 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 34: conflicting requests 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 35: conflicting requests 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 36: conflicting requests 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 37: conflicting requests 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.880 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 39: conflicting requests 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:02:46.881 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.882 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:02:46.883 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:02:47.307 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T02:02:47.307 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T02:02:47.426 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-06T02:02:47.433 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-06T02:02:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-04-06T02:02:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:02:47.435 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 76 M 2024-04-06T02:02:47.435 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 123 M 2024-04-06T02:02:47.435 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-06T02:02:47.676 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 642 kB/s | 66 kB 00:00 2024-04-06T02:02:48.102 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 28 MB/s | 15 MB 00:00 2024-04-06T02:02:48.452 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 35 MB/s | 27 MB 00:00 2024-04-06T02:02:48.760 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-04-06T02:02:48.761 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:02:48.761 INFO:teuthology.orchestra.run.smithi104.stdout:Total 57 MB/s | 76 MB 00:01 2024-04-06T02:02:49.433 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-06T02:02:49.809 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-06T02:02:49.809 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-06T02:02:50.815 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-06T02:02:50.815 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-06T02:02:53.285 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-06T02:02:53.917 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:02:54.026 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:02:56.784 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:02:56.935 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:02:58.371 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:15 2024-04-06T02:03:01.611 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:03:01.717 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:03.645 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr:Modular dependency problems: 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 1: conflicting requests 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 2: conflicting requests 2024-04-06T02:03:05.858 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 3: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 4: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 5: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 6: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 7: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 8: conflicting requests 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.859 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 9: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 10: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 11: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 12: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 13: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 14: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 15: conflicting requests 2024-04-06T02:03:05.860 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 16: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 17: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 18: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 19: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 20: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 21: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 22: conflicting requests 2024-04-06T02:03:05.861 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 23: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 24: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 25: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 26: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 27: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 28: conflicting requests 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.862 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 29: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 30: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 31: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 32: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 33: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 34: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 35: conflicting requests 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.863 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 36: conflicting requests 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 37: conflicting requests 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 39: conflicting requests 2024-04-06T02:03:05.864 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.865 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.866 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:03:05.867 INFO:teuthology.orchestra.run.smithi017.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:03:06.303 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T02:03:06.303 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T02:03:06.401 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:06.407 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-04-06T02:03:06.413 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-04-06T02:03:06.414 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:06.415 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 76 M 2024-04-06T02:03:06.415 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 123 M 2024-04-06T02:03:06.415 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-04-06T02:03:06.797 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 593 kB/s | 66 kB 00:00 2024-04-06T02:03:07.139 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 33 MB/s | 15 MB 00:00 2024-04-06T02:03:07.558 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 38 MB/s | 33 MB 00:00 2024-04-06T02:03:07.806 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 27 MB/s | 27 MB 00:01 2024-04-06T02:03:07.807 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:03:07.807 INFO:teuthology.orchestra.run.smithi017.stdout:Total 55 MB/s | 76 MB 00:01 2024-04-06T02:03:08.505 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-04-06T02:03:08.950 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-04-06T02:03:08.950 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-04-06T02:03:09.978 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-04-06T02:03:09.978 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-04-06T02:03:12.586 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-04-06T02:03:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:03:13.391 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:03:16.316 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:03:16.450 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:03:21.552 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:03:21.592 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:26.735 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:35.739 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:35.739 INFO:teuthology.orchestra.run.smithi104.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T02:03:35.739 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:03:35.784 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:03:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:03:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:03:36.561 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:03:36.562 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-06T02:03:36.883 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:03:36.984 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:36.984 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:03:36.985 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T02:03:36.985 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:03:36.985 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:03:36.985 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-06T02:03:36.985 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:03:36.985 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:03:37.064 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:37.065 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:03:37.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-04-06T02:03:37.098 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-04-06T02:03:37.098 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-04-06T02:03:37.098 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-04-06T02:03:37.098 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 68.el9 2024-04-06T02:03:37.098 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8264254 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:03:37.099 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T02:03:37.100 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T02:03:37.100 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T02:03:37.100 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-04-06T02:03:37.100 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:03:37.100 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T02:03:37.101 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T02:03:37.101 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-432.el9.x86_64 2024-04-06T02:03:37.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T02:03:37.722 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-04-06T02:03:39.191 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T02:03:39.225 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-04-06T02:03:39.229 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-04-06T02:03:39.245 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.gtInybKhNx 2024-04-06T02:03:39.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gtInybKhNx 2024-04-06T02:03:39.314 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.gtInybKhNx 2024-04-06T02:03:39.451 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.gtInybKhNx is 6KB 2024-04-06T02:03:39.499 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.gtInybKhNx 2024-04-06T02:03:39.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T02:03:39.578 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T02:03:39.578 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T02:03:39.578 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T02:03:39.578 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T02:03:39.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T02:03:39.737 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-04-06T02:03:57.880 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:57.880 INFO:teuthology.orchestra.run.smithi017.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T02:03:57.880 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:57.926 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:58.317 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:58.317 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:03:58.317 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:03:58.317 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T02:03:58.758 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T02:03:58.759 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:58.759 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-04-06T02:03:59.092 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:03:59.211 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:59.211 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:03:59.211 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T02:03:59.211 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:03:59.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:03:59.211 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-06T02:03:59.211 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:03:59.211 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:03:59.294 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:03:59.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:03:59.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 68.el9 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8264254 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T02:03:59.332 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:03:59.333 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T02:03:59.334 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T02:03:59.334 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-432.el9.x86_64 2024-04-06T02:03:59.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T02:03:59.938 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-04-06T02:04:01.376 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T02:04:01.403 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-04-06T02:04:01.405 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-04-06T02:04:01.420 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.K1SQZvNPuO 2024-04-06T02:04:01.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K1SQZvNPuO 2024-04-06T02:04:01.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.K1SQZvNPuO 2024-04-06T02:04:01.639 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.K1SQZvNPuO is 6KB 2024-04-06T02:04:01.688 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.K1SQZvNPuO 2024-04-06T02:04:01.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T02:04:01.780 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T02:04:01.781 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T02:04:01.781 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T02:04:01.781 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T02:04:01.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T02:04:01.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-04-06T02:04:09.746 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-06T02:04:09.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:04:09.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:04:28.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-06T02:04:31.936 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-06T02:04:31.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:04:31.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:04:37.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:04:37.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:04:40.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-06T02:04:50.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-04-06T02:04:52.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:04:52.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:04:59.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:04:59.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:05:10.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-06T02:05:25.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:05:25.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:05:25.893 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-06T02:05:26.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:05:26.332 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-04-06T02:05:26.332 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-06T02:05:26.346 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-432.el9.x86_64 2024-04-06T02:05:26.346 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T02:05:26.346 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:05:26.346 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T02:05:27.347 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-06T02:05:27.347 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:05:27.421 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-04-06T02:05:27.461 DEBUG:teuthology.parallel:result is None 2024-04-06T02:05:59.346 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T02:06:11.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:06:11.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:06:11.686 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-04-06T02:06:12.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:06:12.093 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-04-06T02:06:12.093 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-04-06T02:06:12.108 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-432.el9.x86_64 2024-04-06T02:06:12.108 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T02:06:12.108 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:06:12.108 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T02:06:13.109 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-06T02:06:13.110 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:06:13.193 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-04-06T02:06:13.231 DEBUG:teuthology.parallel:result is None 2024-04-06T02:06:13.231 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T02:06:13.242 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T02:06:13.242 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:06:13.247 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:06:13.266 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T02:06:13.274 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T02:06:13.286 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T02:06:13.286 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:06:13.306 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:06:13.494 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T02:06:13.501 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T02:06:13.501 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:06:13.504 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:06:13.540 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:06:13.542 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:06:13.554 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:06:13.556 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:06:13.558 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T02:06:13.565 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T02:06:13.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:06:13.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:06:13.631 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T02:06:13.642 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T02:06:13.642 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:06:13.670 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:06:13.687 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:06:13.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:06:13.814 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:06:13.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:06:13.940 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:06:13.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:06:14.009 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:06:14.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:06:14.106 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:06:14.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:06:14.235 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:06:14.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:06:14.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-04-06T02:06:14.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-04-06T02:06:14.346 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:06:14.395 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:06:14.779 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T02:06:14.787 INFO:teuthology.task.internal:Starting timer... 2024-04-06T02:06:14.787 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T02:06:14.798 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T02:06:14.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-04-06T02:06:14.871 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-04-06T02:06:14.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-04-06T02:06:14.929 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-04-06T02:06:14.931 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T02:06:14.931 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-04-06T02:06:14.956 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-04-06T02:06:14.956 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-04-06T02:06:14.983 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-04-06T02:06:14.983 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-04-06T02:06:14.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T02:06:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:06:15.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T02:06:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:06:15.057 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T02:06:15.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-04-06T02:06:15.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-04-06T02:06:15.152 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T02:06:15.162 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T02:06:15.397 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T02:06:15.399 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T02:10:36.840 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-04-06T02:10:36.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:10:36.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:10:36.933 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-04-06T02:10:37.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-04-06T02:10:37.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:10:37.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:10:37.111 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-06T02:10:37.202 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-06T02:10:37.203 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T02:10:37.213 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T02:10:37.213 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T02:10:37.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:10:37.216 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T02:10:37.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:10:37.247 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T02:10:37.264 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T02:10:37.276 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T02:10:37.294 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T02:10:37.294 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-04-06T02:10:37.308 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-04-06T02:10:37.324 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-04-06T02:10:37.325 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T02:10:37.338 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-04-06T02:10:37.341 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T02:10:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T02:10:37.371 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T02:10:37.391 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-04-06T02:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.394 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.395 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:10:37.423 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:10:37.424 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-06T02:10:37.424 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.424 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.424 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.424 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:10:37.425 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-06T02:10:37.517 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'} 2024-04-06T02:10:37.517 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-06T02:10:37.517 INFO:tasks.cephadm:Cluster fsid is dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:10:37.518 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-06T02:10:37.518 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.17', 'mon.c': '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]', 'mon.b': '172.21.15.104'} 2024-04-06T02:10:37.518 INFO:tasks.cephadm:First mon is mon.a on smithi017 2024-04-06T02:10:37.518 INFO:tasks.cephadm:First mgr is y 2024-04-06T02:10:37.518 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-06T02:10:37.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2024-04-06T02:10:37.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-04-06T02:10:37.574 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-06T02:10:37.575 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:37.857 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 6 02:10 /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:37.857 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:38.131 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 6 02:10 /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:38.131 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:38.147 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T02:10:38.174 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-06T02:10:38.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-06T02:10:38.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-06T02:10:39.033 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T02:10:39.101 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T02:10:57.901 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:10:57.901 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-06T02:10:57.901 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-06T02:10:57.901 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2024-04-06T02:10:57.901 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-06T02:10:57.902 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-06T02:10:57.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-04-06T02:10:57.902 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-06T02:11:00.147 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-06T02:11:00.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2024-04-06T02:11:00.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-04-06T02:11:00.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2024-04-06T02:11:00.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-04-06T02:11:00.297 INFO:tasks.cephadm:Writing seed config... 2024-04-06T02:11:00.298 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-06T02:11:00.298 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-06T02:11:00.298 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-06T02:11:00.299 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-06T02:11:00.300 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:11:00.300 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-06T02:11:00.327 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dbfad14a-f3ba-11ee-b649-cb9ed24678a4 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-06T02:11:00.328 DEBUG:teuthology.orchestra.run.smithi017:mon.a> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service 2024-04-06T02:11:00.370 DEBUG:teuthology.orchestra.run.smithi017:mgr.y> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service 2024-04-06T02:11:00.413 INFO:tasks.cephadm:Bootstrapping... 2024-04-06T02:11:00.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:11:00.616 INFO:teuthology.orchestra.run.smithi017.stderr:-------------------------------------------------------------------------------- 2024-04-06T02:11:00.617 INFO:teuthology.orchestra.run.smithi017.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'dbfad14a-f3ba-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.17', '--skip-admin-label'] 2024-04-06T02:11:00.640 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-04-06T02:11:00.642 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying podman|docker is present... 2024-04-06T02:11:00.664 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-04-06T02:11:00.666 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying lvm2 is present... 2024-04-06T02:11:00.666 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying time synchronization is in place... 2024-04-06T02:11:00.673 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T02:11:00.679 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:11:00.685 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-04-06T02:11:00.691 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-04-06T02:11:00.692 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-04-06T02:11:00.692 INFO:teuthology.orchestra.run.smithi017.stderr:Repeating the final host check... 2024-04-06T02:11:00.712 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-04-06T02:11:00.714 INFO:teuthology.orchestra.run.smithi017.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-06T02:11:00.715 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl is present 2024-04-06T02:11:00.715 INFO:teuthology.orchestra.run.smithi017.stderr:lvcreate is present 2024-04-06T02:11:00.720 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T02:11:00.726 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:11:00.733 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-04-06T02:11:00.738 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-04-06T02:11:00.739 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-04-06T02:11:00.739 INFO:teuthology.orchestra.run.smithi017.stderr:Host looks OK 2024-04-06T02:11:00.739 INFO:teuthology.orchestra.run.smithi017.stderr:Cluster fsid: dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:00.739 INFO:teuthology.orchestra.run.smithi017.stderr:Acquiring lock 139665411653104 on /run/cephadm/dbfad14a-f3ba-11ee-b649-cb9ed24678a4.lock 2024-04-06T02:11:00.740 INFO:teuthology.orchestra.run.smithi017.stderr:Lock 139665411653104 acquired on /run/cephadm/dbfad14a-f3ba-11ee-b649-cb9ed24678a4.lock 2024-04-06T02:11:00.740 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 3300 ... 2024-04-06T02:11:00.740 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 6789 ... 2024-04-06T02:11:00.740 INFO:teuthology.orchestra.run.smithi017.stderr:Base mon IP is 172.21.15.17, final addrv is [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-04-06T02:11:00.743 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.17 metric 100 2024-04-06T02:11:00.743 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.17 metric 100 2024-04-06T02:11:00.745 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-06T02:11:00.745 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-06T02:11:00.745 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-06T02:11:00.748 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-06T02:11:00.748 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-06T02:11:00.749 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-06T02:11:00.749 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-06T02:11:00.749 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:d992/64 scope link noprefixroute 2024-04-06T02:11:00.749 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-06T02:11:00.750 INFO:teuthology.orchestra.run.smithi017.stderr:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2024-04-06T02:11:00.750 INFO:teuthology.orchestra.run.smithi017.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-06T02:11:00.750 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-06T02:11:00.852 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Getting image source signatures 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-06T02:11:01.256 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-06T02:11:01.387 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Writing manifest to image destination 2024-04-06T02:11:01.508 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-06T02:11:01.951 INFO:teuthology.orchestra.run.smithi017.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-06T02:11:02.506 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-06T02:11:02.506 INFO:teuthology.orchestra.run.smithi017.stderr:Extracting ceph user uid/gid from container image... 2024-04-06T02:11:02.923 INFO:teuthology.orchestra.run.smithi017.stderr:stat: 167 167 2024-04-06T02:11:03.488 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial keys... 2024-04-06T02:11:03.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQA3rxBmspd6LxAA0/UHrqVnQ3PafcbB5vNilw== 2024-04-06T02:11:04.767 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQA4rxBm529xKhAAZY9dE481Jsgcn9iuTeoxyw== 2024-04-06T02:11:07.887 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQA7rxBmoIuVBBAA/7K5e5TgA5u3KRlAOeZg5Q== 2024-04-06T02:11:13.581 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial monmap... 2024-04-06T02:11:16.954 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T02:11:16.955 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-06T02:11:16.955 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:16.955 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T02:11:17.429 INFO:teuthology.orchestra.run.smithi017.stderr:monmaptool for a [v2:172.21.15.17:3300,v1:172.21.15.17:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T02:11:17.429 INFO:teuthology.orchestra.run.smithi017.stderr:setting min_mon_release = octopus 2024-04-06T02:11:17.429 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: set fsid to dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:17.429 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T02:11:17.430 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:11:17.430 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mon... 2024-04-06T02:11:18.367 INFO:teuthology.orchestra.run.smithi017.stderr:create mon.a on 2024-04-06T02:11:18.582 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-06T02:11:18.729 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target. 2024-04-06T02:11:18.729 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target. 2024-04-06T02:11:19.278 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service: Unit ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service not loaded. 2024-04-06T02:11:19.285 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target.wants/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@.service. 2024-04-06T02:11:20.015 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-04-06T02:11:20.022 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:11:20.022 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-04-06T02:11:20.022 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-04-06T02:11:20.023 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon to start... 2024-04-06T02:11:20.023 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon... 2024-04-06T02:11:20.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:20 smithi017 ceph-mon[24486]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: cluster: 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: id: dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: services: 2024-04-06T02:11:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.535648s) 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: data: 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pgs: 2024-04-06T02:11:20.543 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: fsmap 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: mgrmap e1: no daemons active 2024-04-06T02:11:21.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:21 smithi017 ceph-mon[24486]: from='client.? 172.21.15.17:0/4291175157' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-06T02:11:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:mon is available 2024-04-06T02:11:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:Assimilating anything we can from ceph.conf... 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-04-06T02:11:21.616 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-06T02:11:21.617 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-06T02:11:21.617 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:21.617 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-04-06T02:11:21.617 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-06T02:11:21.617 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-06T02:11:22.142 INFO:teuthology.orchestra.run.smithi017.stderr:Generating new minimal ceph.conf... 2024-04-06T02:11:22.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:22 smithi017 ceph-mon[24486]: from='client.? 172.21.15.17:0/1130915933' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T02:11:22.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:22 smithi017 ceph-mon[24486]: from='client.? 172.21.15.17:0/1130915933' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T02:11:23.208 INFO:teuthology.orchestra.run.smithi017.stderr:Restarting the monitor... 2024-04-06T02:11:23.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 ceph-mon[24486]: from='client.? 172.21.15.17:0/775394882' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:11:23.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 systemd[1]: Stopping Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:11:23.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 bash[24719]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-06T02:11:23.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[24482]: 2024-04-06T02:11:23.368+0000 7f4b0bf4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:11:23.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[24482]: 2024-04-06T02:11:23.368+0000 7f4b0bf4b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T02:11:23.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 podman[24730]: 2024-04-06 02:11:23.512394858 +0000 UTC m=+0.239336582 container died 59939e427960f6916461b71cb5fafdf16157bc81f6b407c1e6b0b9cb39cd5b17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True) 2024-04-06T02:11:23.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 podman[24730]: 2024-04-06 02:11:23.837433618 +0000 UTC m=+0.564375355 container cleanup 59939e427960f6916461b71cb5fafdf16157bc81f6b407c1e6b0b9cb39cd5b17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-06T02:11:23.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 bash[24730]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a 2024-04-06T02:11:23.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:23 smithi017 bash[24756]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon.a" found: no such container 2024-04-06T02:11:24.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 podman[24744]: 2024-04-06 02:11:24.030005477 +0000 UTC m=+0.510014952 container remove 59939e427960f6916461b71cb5fafdf16157bc81f6b407c1e6b0b9cb39cd5b17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git, distribution-scope=public, version=8, ceph=True, release=754, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T02:11:24.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-06T02:11:24.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 systemd[1]: Stopped Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:11:24.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 systemd[1]: Starting Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:11:24.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 podman[24826]: 2024-04-06 02:11:24.239060624 +0000 UTC m=+0.092870861 container create 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, architecture=x86_64) 2024-04-06T02:11:24.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 podman[24826]: 2024-04-06 02:11:24.164472036 +0000 UTC m=+0.018282232 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-06T02:11:24.474 INFO:teuthology.orchestra.run.smithi017.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 podman[24826]: 2024-04-06 02:11:24.43302803 +0000 UTC m=+0.286838226 container init 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, release=754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, RELEASE=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 podman[24826]: 2024-04-06 02:11:24.4358807 +0000 UTC m=+0.289690895 container start 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=) 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: pidfile_write: ignore empty --pid-file 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: load: jerasure load: lrc 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: RocksDB version: 6.15.5 2024-04-06T02:11:24.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-06T02:11:24.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Compile date Apr 18 2022 2024-04-06T02:11:24.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: DB SUMMARY 2024-04-06T02:11:24.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: DB Session ID: V0SIAL11TMP41151MF2D 2024-04-06T02:11:24.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:11:24.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82633 ; 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.env: 0x5592100c6860 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.fs: Posix File System 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.info_log: 0x55921163ddc0 2024-04-06T02:11:24.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.statistics: (nil) 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.use_fsync: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:11:24.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.db_log_dir: 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.write_buffer_manager: 0x55921172e2a0 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:11:24.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.unordered_write: 0 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.row_cache: None 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.wal_filter: None 2024-04-06T02:11:24.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.preserve_deletes: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:11:24.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_open_files: -1 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:11:24.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Compression algorithms supported: 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kZSTD supported: 0 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:11:24.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.merge_operator: 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_filter: None 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559211609d00) 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-04-06T02:11:24.607 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: hash_index_allow_collision: 1 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 1 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x559211674170 2024-04-06T02:11:24.608 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-04-06T02:11:24.609 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 4 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-04-06T02:11:24.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression: NoCompression 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.num_levels: 7 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:11:24.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:11:24.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:11:24.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:11:24.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:11:24.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:11:24.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:11:24.613 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:11:24.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.arena_block_size: 4194304 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:11:24.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:11:24.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:11:24.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:11:24.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.ttl: 2592000 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T02:11:24.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369484462646, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369484463342, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79183, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77446, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71755, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712369484, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f4ba718f-42c7-4d55-8d31-52af9659ddcc", "db_session_id": "V0SIAL11TMP41151MF2D"}} 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369484463681, "job": 1, "event": "recovery_finished"} 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55921165aa80 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: DB pointer 0x55921166a000 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-06T02:11:24.620 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.621 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:11:24.622 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 25.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 25.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.623 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:11:24.624 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 25.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???) e1 preinit fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).mds e1 new map 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).mds e1 print_map 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-04-06T02:11:24.625 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expand map: {default=false} 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta from 'false' to 'false' 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expanded map: {default=false} 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expand map: {default=info} 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta from 'info' to 'info' 2024-04-06T02:11:24.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expanded map: {default=info} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expand map: {default=daemon} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expanded map: {default=daemon} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expand map: {default=debug} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta from 'debug' to 'debug' 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: expand_channel_meta expanded map: {default=debug} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 bash[24826]: 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 systemd[1]: Started Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: fsmap 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T02:11:24.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:24 smithi017 ceph-mon[24844]: mgrmap e1: no daemons active 2024-04-06T02:11:25.590 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-06T02:11:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:11:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mgr... 2024-04-06T02:11:25.591 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying port 9283 ... 2024-04-06T02:11:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service: Unit ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service not loaded. 2024-04-06T02:11:25.770 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4.target.wants/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service → /etc/systemd/system/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@.service. 2024-04-06T02:11:26.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:25 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3639095814' entity='client.admin' 2024-04-06T02:11:26.429 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-04-06T02:11:26.436 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:11:26.437 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-04-06T02:11:26.437 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-04-06T02:11:26.446 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-04-06T02:11:26.453 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:11:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-04-06T02:11:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-06T02:11:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr to start... 2024-04-06T02:11:26.455 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr... 2024-04-06T02:11:26.574 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:26 smithi017 systemd[1]: Started Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "dbfad14a-f3ba-11ee-b649-cb9ed24678a4", 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T02:11:27.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T02:11:27.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:27.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T02:11:27.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T02:11:27.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T02:11:27.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T02:11:27.005 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T02:11:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-04-06T02:11:20.016424+0000", 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T02:11:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:11:27.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1376491775' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:11:27.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:26.837+0000 7f112c73a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:11:27.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:26.931+0000 7f112c73a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:11:27.868 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:27.554+0000 7f112c73a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:11:27.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:27.644+0000 7f112c73a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:11:27.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:27.729+0000 7f112c73a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:11:27.984 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (1/15)... 2024-04-06T02:11:28.148 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:27.879+0000 7f112c73a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:11:28.148 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:27.961+0000 7f112c73a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:11:28.148 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:28.074+0000 7f112c73a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:11:28.148 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:28.158+0000 7f112c73a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:11:28.398 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:28.322+0000 7f112c73a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:11:28.398 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:28.408+0000 7f112c73a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:11:29.589 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:29 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:29.468+0000 7f112c73a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:11:29.589 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:29 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:29.600+0000 7f112c73a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:11:29.873 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:29 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:29.795+0000 7f112c73a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:11:29.873 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:29 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:29.885+0000 7f112c73a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:11:30.148 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:30.026+0000 7f112c73a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:11:30.434 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:30.352+0000 7f112c73a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "dbfad14a-f3ba-11ee-b649-cb9ed24678a4", 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.539 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T02:11:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T02:11:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T02:11:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T02:11:30.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T02:11:30.544 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T02:11:30.544 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T02:11:30.545 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T02:11:30.546 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:30.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-04-06T02:11:20.016424+0000", 2024-04-06T02:11:30.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:30.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:30.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T02:11:30.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:11:30.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2858436012' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:11:30.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:30.584+0000 7f112c73a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:11:30.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:30.670+0000 7f112c73a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:11:31.129 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:30.845+0000 7f112c73a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:11:31.131 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (2/15)... 2024-04-06T02:11:32.076 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:31 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:31.786+0000 7f112c73a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:11:32.077 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:32.088+0000 7f112c73a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:11:32.327 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:32.185+0000 7f112c73a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:11:32.328 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:32.339+0000 7f112c73a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: Activating manager daemon y 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: mgrmap e2: y(active, starting, since 0.00257523s) 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:11:32.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: Manager daemon y is now available 2024-04-06T02:11:32.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:11:32.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:11:32.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' 2024-04-06T02:11:32.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' 2024-04-06T02:11:32.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:32 smithi017 ceph-mon[24844]: from='mgr.14100 172.21.15.17:0/999321328' entity='mgr.y' 2024-04-06T02:11:33.808 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "dbfad14a-f3ba-11ee-b649-cb9ed24678a4", 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-04-06T02:11:33.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-06T02:11:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-06T02:11:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-06T02:11:33.811 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-06T02:11:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-06T02:11:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-06T02:11:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-04-06T02:11:33.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.815 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-04-06T02:11:20.016424+0000", 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-06T02:11:33.816 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:11:34.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:34 smithi017 ceph-mon[24844]: mgrmap e3: y(active, since 1.00639s) 2024-04-06T02:11:34.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:34 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1860215482' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:11:34.379 INFO:teuthology.orchestra.run.smithi017.stderr:mgr is available 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-06T02:11:34.926 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-06T02:11:34.927 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-06T02:11:35.444 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling cephadm module... 2024-04-06T02:11:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:35 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/841037142' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T02:11:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:35 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/841037142' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T02:11:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:35 smithi017 ceph-mon[24844]: mgrmap e4: y(active, since 3s) 2024-04-06T02:11:36.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:36 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2693054671' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-06T02:11:36.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setuser ceph since I am not root 2024-04-06T02:11:36.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setgroup ceph since I am not root 2024-04-06T02:11:37.031 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:36.979+0000 7ff8caac0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:11:37.334 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:37.074+0000 7ff8caac0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:11:37.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2693054671' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-06T02:11:37.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-mon[24844]: mgrmap e5: y(active, since 4s) 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-06T02:11:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:11:37.962 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:37.692+0000 7ff8caac0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:11:37.962 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:37.785+0000 7ff8caac0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:11:37.962 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:37 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:37.869+0000 7ff8caac0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:11:38.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.022+0000 7ff8caac0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:11:38.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.104+0000 7ff8caac0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:11:38.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.215+0000 7ff8caac0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:11:38.282 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-04-06T02:11:38.282 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 5... 2024-04-06T02:11:38.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/590266917' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T02:11:38.551 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.301+0000 7ff8caac0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:11:38.551 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.471+0000 7ff8caac0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:11:38.551 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:38.562+0000 7ff8caac0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:11:39.934 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:39 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:39.617+0000 7ff8caac0000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:11:39.934 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:39 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:39.750+0000 7ff8caac0000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:11:39.934 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:39 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:39.945+0000 7ff8caac0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:11:40.242 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.034+0000 7ff8caac0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:11:40.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.175+0000 7ff8caac0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:11:40.742 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.499+0000 7ff8caac0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:11:40.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.727+0000 7ff8caac0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:11:41.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.814+0000 7ff8caac0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:11:41.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:40 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:40.988+0000 7ff8caac0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:11:42.218 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:41.926+0000 7ff8caac0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:11:42.219 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:42.230+0000 7ff8caac0000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:11:42.472 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:42.329+0000 7ff8caac0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:11:42.742 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:42.484+0000 7ff8caac0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-06T02:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-04-06T02:11:43.547 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:11:43] ENGINE Bus STARTING 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:11:43] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:11:43] ENGINE Bus STARTED 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: Active manager daemon y restarted 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: Activating manager daemon y 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: osdmap e2: 0 total, 0 up, 0 in 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: mgrmap e6: y(active, starting, since 0.0524394s) 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:11:43.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: Manager daemon y is now available 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:11:43.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:43 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:11:44.071 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 5 is available 2024-04-06T02:11:44.072 INFO:teuthology.orchestra.run.smithi017.stderr:Setting orchestrator backend to cephadm... 2024-04-06T02:11:44.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:44 smithi017 ceph-mon[24844]: [06/Apr/2024:02:11:43] ENGINE Bus STARTING 2024-04-06T02:11:44.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:44 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:44.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:44 smithi017 ceph-mon[24844]: mgrmap e7: y(active, since 1.05502s) 2024-04-06T02:11:44.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:44 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:45.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: [06/Apr/2024:02:11:43] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:11:45.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: [06/Apr/2024:02:11:43] ENGINE Bus STARTED 2024-04-06T02:11:45.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T02:11:45.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T02:11:45.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:45.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:45 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:45.860 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: value unchanged 2024-04-06T02:11:46.399 INFO:teuthology.orchestra.run.smithi017.stderr:Generating ssh key... 2024-04-06T02:11:46.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:46 smithi017 ceph-mon[24844]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:46.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:46 smithi017 ceph-mon[24844]: mgrmap e8: y(active, since 3s) 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-mon[24844]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: Generating public/private rsa key pair. 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: Your identification has been saved in /tmp/tmplspkvxlu/key. 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: Your public key has been saved in /tmp/tmplspkvxlu/key.pub. 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: The key fingerprint is: 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: SHA256:EBRRai+XglmPq2MsGnqvsYfLbscT/40IvzsxwPoW0WE ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:47.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: The key's randomart image is: 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: +---[RSA 3072]----+ 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | .=+. | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | Eo | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | . o=. | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | +=.= . | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | .oo+ S | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: | . o o= | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: |. .=.+.o | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: |.o=o@+o. o | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: |o=*Xoo==o . | 2024-04-06T02:11:47.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: +----[SHA256]-----+ 2024-04-06T02:11:48.511 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMBU+1xUbCdKPA8f4XDzF4NCc+sJMbAG3ATxPBa51BBILkQ4eAtsou8nu7wxVmN6koSQGTHX3gkeQyEqcmw6pjoGICsfhABlXl7SEJaOQ0rXf7v/NnHPtlZ7Sok9XLdS686JEFKyV1pIDPmzm+gGBAhwQx7tXCnI4CgVie10WLVPEOzkiw6qNKXKirEHYFYDZXjNWLTvpsSyuFMys2y2Ni/K8j4K1ndYFYlAdYfjBPHalriUU9Q6d4b7YGIFp/ZzYhCXavS9iwEpppUoj0GLkDCp5XDJPzQ1knZ/5lkZHxRYqw7Q1y3Xx1OQ6C3huxw2RPVJMi6Vmo9hLfaUODvAo7VTCzlXAXNLoI+mWC47GnwPAlXgX24T5ZWgB5X4ARsANPgnjyPN7rfaUO+Fo7GwbTB5zFzUsJXzR1KSMRxNaGnKjSkkxlzPXFAEEurs4plv2laYMf2RxkV9dFQc0OwvI1a5WlAzs3GyYNcdU6j1bZjV5vZ8dEd9gBejGv/g95sZ0= ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:11:48.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:48 smithi017 ceph-mon[24844]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:48.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:48 smithi017 ceph-mon[24844]: Generating ssh key... 2024-04-06T02:11:49.106 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-06T02:11:49.106 INFO:teuthology.orchestra.run.smithi017.stderr:Adding key to root@localhost authorized_keys... 2024-04-06T02:11:49.106 INFO:teuthology.orchestra.run.smithi017.stderr:Adding host smithi017... 2024-04-06T02:11:50.674 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Added host 'smithi017' with addr '172.21.15.17' 2024-04-06T02:11:50.685 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:50 smithi017 ceph-mon[24844]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:51.236 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mon service... 2024-04-06T02:11:51.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:51 smithi017 ceph-mon[24844]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:51.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:51 smithi017 ceph-mon[24844]: Deploying cephadm binary to smithi017 2024-04-06T02:11:51.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:51 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:51.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:51 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:11:51.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-06T02:11:52.427 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mgr service... 2024-04-06T02:11:52.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:52 smithi017 ceph-mon[24844]: Added host smithi017 2024-04-06T02:11:52.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:52 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:53.171 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-06T02:11:53.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:53 smithi017 ceph-mon[24844]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:53.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:53 smithi017 ceph-mon[24844]: Saving service mon spec with placement count:5 2024-04-06T02:11:53.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:53 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:54.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:54 smithi017 ceph-mon[24844]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:11:54.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:54 smithi017 ceph-mon[24844]: Saving service mgr spec with placement count:2 2024-04-06T02:11:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:55 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:55 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:55.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:55 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2168097547' entity='client.admin' 2024-04-06T02:11:57.342 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling the dashboard module... 2024-04-06T02:11:57.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:57 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2132663497' entity='client.admin' 2024-04-06T02:11:57.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:57 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:58.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:58 smithi017 ceph-mon[24844]: from='mgr.14118 172.21.15.17:0/1378828117' entity='mgr.y' 2024-04-06T02:11:58.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:58 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3656952692' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-06T02:11:58.832 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setuser ceph since I am not root 2024-04-06T02:11:58.832 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setgroup ceph since I am not root 2024-04-06T02:11:59.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:59.153+0000 7fee905cf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:11:59.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:59.244+0000 7fee905cf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:11:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3656952692' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-06T02:11:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-mon[24844]: mgrmap e9: y(active, since 16s) 2024-04-06T02:11:59.992 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:59.861+0000 7fee905cf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:11:59.992 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:11:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:11:59.954+0000 7fee905cf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:12:00.189 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:12:00.189 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-06T02:12:00.190 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-04-06T02:12:00.190 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-06T02:12:00.190 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-06T02:12:00.190 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:12:00.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.044+0000 7fee905cf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:12:00.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.195+0000 7fee905cf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:12:00.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.281+0000 7fee905cf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:12:00.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.392+0000 7fee905cf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:12:00.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3865796886' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T02:12:00.633 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.479+0000 7fee905cf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:12:00.633 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.643+0000 7fee905cf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:12:00.803 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-04-06T02:12:00.803 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 9... 2024-04-06T02:12:00.925 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:00.730+0000 7fee905cf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:12:02.103 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:01.787+0000 7fee905cf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:12:02.103 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:01.919+0000 7fee905cf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:12:02.103 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.113+0000 7fee905cf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:12:02.492 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.202+0000 7fee905cf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:12:02.492 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.342+0000 7fee905cf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:12:02.965 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.665+0000 7fee905cf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:12:02.965 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.889+0000 7fee905cf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:12:02.965 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:02.976+0000 7fee905cf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:12:03.242 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:03.149+0000 7fee905cf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:12:04.374 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:04.083+0000 7fee905cf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:12:04.374 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:04.385+0000 7fee905cf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:12:04.624 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:04.482+0000 7fee905cf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:12:04.624 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:12:04.635+0000 7fee905cf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:12:05.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: Active manager daemon y restarted 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: Activating manager daemon y 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: osdmap e3: 0 total, 0 up, 0 in 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: mgrmap e10: y(active, starting, since 0.828217s) 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: Manager daemon y is now available 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:12:05.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:12:06.473 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:12:06] ENGINE Bus STARTING 2024-04-06T02:12:06.474 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-04-06T02:12:06.474 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-06T02:12:06.474 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-04-06T02:12:06.474 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-04-06T02:12:06.742 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:12:06] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:12:06.742 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:12:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:12:06] ENGINE Bus STARTED 2024-04-06T02:12:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 9 is available 2024-04-06T02:12:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:Generating a dashboard self-signed certificate... 2024-04-06T02:12:07.590 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: [06/Apr/2024:02:12:06] ENGINE Bus STARTING 2024-04-06T02:12:07.590 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T02:12:07.590 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: mgrmap e11: y(active, since 1.83003s) 2024-04-06T02:12:07.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T02:12:07.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: [06/Apr/2024:02:12:06] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:12:07.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: [06/Apr/2024:02:12:06] ENGINE Bus STARTED 2024-04-06T02:12:07.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:07.728 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-06T02:12:08.312 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial admin user... 2024-04-06T02:12:08.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:08.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: mgrmap e12: y(active, since 2s) 2024-04-06T02:12:08.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:08.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:08.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:08.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:09.212 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$4LM3W.PTEDyPwbw57Yg/U.3DGkuZebDyfqhQlSEl3DWlbrA6jOqee", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712369529, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-06T02:12:10.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:10 smithi017 ceph-mon[24844]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:10.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:10 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:10.485 INFO:teuthology.orchestra.run.smithi017.stderr:Fetching dashboard port number... 2024-04-06T02:12:11.153 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 8443 2024-04-06T02:12:11.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:11 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/184006069' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-06T02:12:11.751 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-04-06T02:12:11.758 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-04-06T02:12:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-04-06T02:12:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-06T02:12:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph Dashboard is now available at: 2024-04-06T02:12:11.761 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:11.761 INFO:teuthology.orchestra.run.smithi017.stderr: URL: https://smithi017.front.sepia.ceph.com:8443/ 2024-04-06T02:12:11.761 INFO:teuthology.orchestra.run.smithi017.stderr: User: admin 2024-04-06T02:12:11.761 INFO:teuthology.orchestra.run.smithi017.stderr: Password: 8f3ualfe9g 2024-04-06T02:12:11.762 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling autotune for osd_memory_target 2024-04-06T02:12:12.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:12 smithi017 ceph-mon[24844]: mgrmap e13: y(active, since 6s) 2024-04-06T02:12:13.735 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr:You can access the Ceph CLI with: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: ceph telemetry on 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.391 INFO:teuthology.orchestra.run.smithi017.stderr:For more information see: 2024-04-06T02:12:14.392 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.392 INFO:teuthology.orchestra.run.smithi017.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-06T02:12:14.392 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:12:14.392 INFO:teuthology.orchestra.run.smithi017.stderr:Bootstrap complete. 2024-04-06T02:12:14.429 INFO:tasks.cephadm:Fetching config... 2024-04-06T02:12:14.430 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:14.430 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-06T02:12:14.446 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-06T02:12:14.446 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:14.446 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-06T02:12:14.501 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-06T02:12:14.501 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:14.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-06T02:12:14.568 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-06T02:12:14.568 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:14.568 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-06T02:12:14.624 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-06T02:12:14.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMBU+1xUbCdKPA8f4XDzF4NCc+sJMbAG3ATxPBa51BBILkQ4eAtsou8nu7wxVmN6koSQGTHX3gkeQyEqcmw6pjoGICsfhABlXl7SEJaOQ0rXf7v/NnHPtlZ7Sok9XLdS686JEFKyV1pIDPmzm+gGBAhwQx7tXCnI4CgVie10WLVPEOzkiw6qNKXKirEHYFYDZXjNWLTvpsSyuFMys2y2Ni/K8j4K1ndYFYlAdYfjBPHalriUU9Q6d4b7YGIFp/ZzYhCXavS9iwEpppUoj0GLkDCp5XDJPzQ1knZ/5lkZHxRYqw7Q1y3Xx1OQ6C3huxw2RPVJMi6Vmo9hLfaUODvAo7VTCzlXAXNLoI+mWC47GnwPAlXgX24T5ZWgB5X4ARsANPgnjyPN7rfaUO+Fo7GwbTB5zFzUsJXzR1KSMRxNaGnKjSkkxlzPXFAEEurs4plv2laYMf2RxkV9dFQc0OwvI1a5WlAzs3GyYNcdU6j1bZjV5vZ8dEd9gBejGv/g95sZ0= ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T02:12:14.716 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMBU+1xUbCdKPA8f4XDzF4NCc+sJMbAG3ATxPBa51BBILkQ4eAtsou8nu7wxVmN6koSQGTHX3gkeQyEqcmw6pjoGICsfhABlXl7SEJaOQ0rXf7v/NnHPtlZ7Sok9XLdS686JEFKyV1pIDPmzm+gGBAhwQx7tXCnI4CgVie10WLVPEOzkiw6qNKXKirEHYFYDZXjNWLTvpsSyuFMys2y2Ni/K8j4K1ndYFYlAdYfjBPHalriUU9Q6d4b7YGIFp/ZzYhCXavS9iwEpppUoj0GLkDCp5XDJPzQ1knZ/5lkZHxRYqw7Q1y3Xx1OQ6C3huxw2RPVJMi6Vmo9hLfaUODvAo7VTCzlXAXNLoI+mWC47GnwPAlXgX24T5ZWgB5X4ARsANPgnjyPN7rfaUO+Fo7GwbTB5zFzUsJXzR1KSMRxNaGnKjSkkxlzPXFAEEurs4plv2laYMf2RxkV9dFQc0OwvI1a5WlAzs3GyYNcdU6j1bZjV5vZ8dEd9gBejGv/g95sZ0= ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:14.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMBU+1xUbCdKPA8f4XDzF4NCc+sJMbAG3ATxPBa51BBILkQ4eAtsou8nu7wxVmN6koSQGTHX3gkeQyEqcmw6pjoGICsfhABlXl7SEJaOQ0rXf7v/NnHPtlZ7Sok9XLdS686JEFKyV1pIDPmzm+gGBAhwQx7tXCnI4CgVie10WLVPEOzkiw6qNKXKirEHYFYDZXjNWLTvpsSyuFMys2y2Ni/K8j4K1ndYFYlAdYfjBPHalriUU9Q6d4b7YGIFp/ZzYhCXavS9iwEpppUoj0GLkDCp5XDJPzQ1knZ/5lkZHxRYqw7Q1y3Xx1OQ6C3huxw2RPVJMi6Vmo9hLfaUODvAo7VTCzlXAXNLoI+mWC47GnwPAlXgX24T5ZWgB5X4ARsANPgnjyPN7rfaUO+Fo7GwbTB5zFzUsJXzR1KSMRxNaGnKjSkkxlzPXFAEEurs4plv2laYMf2RxkV9dFQc0OwvI1a5WlAzs3GyYNcdU6j1bZjV5vZ8dEd9gBejGv/g95sZ0= ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T02:12:14.779 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMBU+1xUbCdKPA8f4XDzF4NCc+sJMbAG3ATxPBa51BBILkQ4eAtsou8nu7wxVmN6koSQGTHX3gkeQyEqcmw6pjoGICsfhABlXl7SEJaOQ0rXf7v/NnHPtlZ7Sok9XLdS686JEFKyV1pIDPmzm+gGBAhwQx7tXCnI4CgVie10WLVPEOzkiw6qNKXKirEHYFYDZXjNWLTvpsSyuFMys2y2Ni/K8j4K1ndYFYlAdYfjBPHalriUU9Q6d4b7YGIFp/ZzYhCXavS9iwEpppUoj0GLkDCp5XDJPzQ1knZ/5lkZHxRYqw7Q1y3Xx1OQ6C3huxw2RPVJMi6Vmo9hLfaUODvAo7VTCzlXAXNLoI+mWC47GnwPAlXgX24T5ZWgB5X4ARsANPgnjyPN7rfaUO+Fo7GwbTB5zFzUsJXzR1KSMRxNaGnKjSkkxlzPXFAEEurs4plv2laYMf2RxkV9dFQc0OwvI1a5WlAzs3GyYNcdU6j1bZjV5vZ8dEd9gBejGv/g95sZ0= ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:14.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-06T02:12:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:14 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1558240436' entity='client.admin' 2024-04-06T02:12:17.728 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-06T02:12:17.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-06T02:12:18.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3373428298' entity='client.admin' 2024-04-06T02:12:18.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:18.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:12:18.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:18.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:18.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:17 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:21.096 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-04-06T02:12:21.096 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:12:21.096 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-04-06T02:12:21.115 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:12:21.115 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:21.172 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-04-06T02:12:21.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi104 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:21.743 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:22 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:23 smithi017 ceph-mon[24844]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:23 smithi017 ceph-mon[24844]: Deploying cephadm binary to smithi104 2024-04-06T02:12:24.029 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-04-06T02:12:24.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-06T02:12:25.232 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: Added host smithi104 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:25.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:12:26.497 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-04-06T02:12:26.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:26 smithi017 ceph-mon[24844]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:26.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:26 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:27.126 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-06T02:12:27.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-06T02:12:27.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:27 smithi017 ceph-mon[24844]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:12:28.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:28 smithi017 ceph-mon[24844]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:28.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:28.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:28.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:12:28.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:29.649 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-04-06T02:12:29.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:29 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/390269381' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T02:12:30.271 INFO:tasks.cephadm:Adding mon.a on smithi017 2024-04-06T02:12:30.271 INFO:tasks.cephadm:Adding mon.c on smithi017 2024-04-06T02:12:30.271 INFO:tasks.cephadm:Adding mon.b on smithi104 2024-04-06T02:12:30.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi104:172.21.15.104=b' 2024-04-06T02:12:30.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:30 smithi017 ceph-mon[24844]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:30.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:30 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/390269381' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T02:12:30.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:30 smithi017 ceph-mon[24844]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:32.076 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-04-06T02:12:32.751 DEBUG:teuthology.orchestra.run.smithi017:mon.c> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.c.service 2024-04-06T02:12:32.753 DEBUG:teuthology.orchestra.run.smithi104:mon.b> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.b.service 2024-04-06T02:12:32.755 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T02:12:32.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:12:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:33.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:33 smithi017 ceph-mon[24844]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi104:172.21.15.104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:33.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:33 smithi017 ceph-mon[24844]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi104:172.21.15.104=b;count:3 2024-04-06T02:12:33.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:33 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:12:33.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:33 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:33.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:33 smithi017 ceph-mon[24844]: Deploying daemon mon.c on smithi017 2024-04-06T02:12:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:12:34.524 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:11:16.126688Z","created":"2024-04-06T02:11:16.126688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:12:34.525 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-04-06T02:12:34.924 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:34 smithi017 ceph-mon[24844]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:34.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:34 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/1170117183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:12:36.014 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T02:12:36.014 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:12:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:41.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: mon.a calling monitor election 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: mon.c calling monitor election 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-06T02:12:41.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: fsmap 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: mgrmap e13: y(active, since 36s) 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: overall HEALTH_OK 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:41.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:41.485 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:12:41.485 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:12:35.824923Z","created":"2024-04-06T02:11:16.126688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T02:12:41.488 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-04-06T02:12:42.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[24844]: Deploying daemon mon.b on smithi104 2024-04-06T02:12:42.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[24844]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:42.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/1684419475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:12:42.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:42.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[29567]: Deploying daemon mon.b on smithi104 2024-04-06T02:12:42.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[29567]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:42.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/1684419475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:12:42.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:41 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:43.452 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T02:12:43.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:12:44.793 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:44 smithi104 systemd[1]: Starting Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:12:44.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:44 smithi017 ceph-mon[29567]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:44.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:44 smithi017 ceph-mon[24844]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:45.689 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 podman[25687]: 2024-04-06 02:12:45.474043226 +0000 UTC m=+0.019090766 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-06T02:12:45.689 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 podman[25687]: 2024-04-06 02:12:45.583264605 +0000 UTC m=+0.128312142 container create 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, ceph=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., distribution-scope=public) 2024-04-06T02:12:45.704 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:12:45.705 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:12:35.824923Z","created":"2024-04-06T02:11:16.126688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T02:12:45.706 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 podman[25687]: 2024-04-06 02:12:45.802285224 +0000 UTC m=+0.347332754 container init 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, maintainer=Guillaume Abrioux , ceph=True, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, name=centos-stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD) 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 podman[25687]: 2024-04-06 02:12:45.805971642 +0000 UTC m=+0.351019173 container start 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , name=centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream) 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: pidfile_write: ignore empty --pid-file 2024-04-06T02:12:46.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: load: jerasure load: lrc 2024-04-06T02:12:46.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: RocksDB version: 6.15.5 2024-04-06T02:12:46.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-06T02:12:46.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Compile date Apr 18 2022 2024-04-06T02:12:46.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: DB SUMMARY 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: DB Session ID: UGIFX0DHSJX2YJYKXIBR 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:12:46.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.env: 0x56179e6c6860 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.fs: Posix File System 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.info_log: 0x5617a04a9dc0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.statistics: (nil) 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.use_fsync: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:12:46.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.db_log_dir: 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:12:46.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.write_buffer_manager: 0x5617a059a240 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:12:46.171 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.unordered_write: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.row_cache: None 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.wal_filter: None 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.preserve_deletes: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:12:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:12:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_open_files: -1 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:12:46.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Compression algorithms supported: 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kZSTD supported: 0 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.merge_operator: 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_filter: None 2024-04-06T02:12:46.175 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617a0475d00) 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_type: 0 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_index_type: 0 2024-04-06T02:12:46.176 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_shortening: 1 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: hash_index_allow_collision: 1 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: checksum: 1 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: no_block_cache: 0 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache: 0x5617a04e0170 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_options: 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: capacity : 536870912 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_shard_bits : 4 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: strict_capacity_limit : 0 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_compressed: (nil) 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: persistent_cache: (nil) 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size: 4096 2024-04-06T02:12:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size_deviation: 10 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_restart_interval: 16 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_block_restart_interval: 1 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: metadata_block_size: 4096 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: partition_filters: 0 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: use_delta_encoding: 1 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: whole_key_filtering: 1 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: verify_compression: 0 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: format_version: 4 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_index_compression: 1 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_align: 0 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:12:46.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression: NoCompression 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.num_levels: 7 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:12:46.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:12:46.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.arena_block_size: 4194304 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:12:46.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:12:46.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-06T02:12:46.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:12:46.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:12:46.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:12:46.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:12:46.183 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:12:46.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.ttl: 2592000 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369565831770, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-06T02:12:46.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369565832078, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712369565, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f44be719-33cf-4c24-a058-097ef520992b", "db_session_id": "UGIFX0DHSJX2YJYKXIBR"}} 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712369565832386, "job": 1, "event": "recovery_finished"} 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5617a04c6a80 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: DB pointer 0x5617a04d6000 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** DB Stats ** 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:12:46.186 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:12:46.187 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:12:46.188 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0.00 0.00 1 0.000 0 0 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:12:46.189 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: starting mon.b rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(???) e0 preinit fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).mds e1 new map 2024-04-06T02:12:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: e1 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: legacy client fscid: -1 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout: No filesystems configured 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-06T02:12:46.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.a calling monitor election 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.c calling monitor election 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:46.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: fsmap 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mgrmap e13: y(active, since 36s) 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: overall HEALTH_OK 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: Deploying daemon mon.b on smithi104 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:46.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/1684419475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 ceph-mon[25739]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 bash[25687]: 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec 2024-04-06T02:12:46.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:45 smithi104 systemd[1]: Started Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:12:47.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T02:12:47.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:12:51.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: mon.a calling monitor election 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: mon.c calling monitor election 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: fsmap 2024-04-06T02:12:51.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: mgrmap e13: y(active, since 46s) 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: mon.a calling monitor election 2024-04-06T02:12:51.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: mon.c calling monitor election 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: fsmap 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: mgrmap e13: y(active, since 46s) 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-06T02:12:51.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-06T02:12:51.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2024-04-06T02:12:51.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:51.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:51.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:51.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:50 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:52.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[24844]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[29567]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:52.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-06T02:12:52.236 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:12:45.851852Z","created":"2024-04-06T02:11:16.126688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T02:12:52.237 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 3 2024-04-06T02:12:52.764 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-06T02:12:52.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-06T02:12:53.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:12:53.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: mon.a calling monitor election 2024-04-06T02:12:53.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: mon.c calling monitor election 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: fsmap 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:53.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: mgrmap e13: y(active, since 46s) 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mon.b calling monitor election 2024-04-06T02:12:53.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mon.b calling monitor election 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mon.c calling monitor election 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mon.a calling monitor election 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: fsmap 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: mgrmap e13: y(active, since 48s) 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: Cluster is now healthy 2024-04-06T02:12:53.130 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:53 smithi104 ceph-mon[25739]: overall HEALTH_OK 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mon.b calling monitor election 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mon.b calling monitor election 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mon.c calling monitor election 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mon.a calling monitor election 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: fsmap 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: mgrmap e13: y(active, since 48s) 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: Cluster is now healthy 2024-04-06T02:12:53.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[24844]: overall HEALTH_OK 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mon.b calling monitor election 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mon.b calling monitor election 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mon.c calling monitor election 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mon.a calling monitor election 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: fsmap 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: mgrmap e13: y(active, since 48s) 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: Cluster is now healthy 2024-04-06T02:12:53.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:53 smithi017 ceph-mon[29567]: overall HEALTH_OK 2024-04-06T02:12:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:54 smithi017 ceph-mon[24844]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:54 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:54.394 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:54 smithi017 ceph-mon[29567]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:54.394 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:54 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:54.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:54 smithi104 ceph-mon[25739]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:54.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:54 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:12:55.066 INFO:teuthology.orchestra.run.smithi017.stdout:# minimal ceph.conf for dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:55.066 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-04-06T02:12:55.066 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:12:55.066 INFO:teuthology.orchestra.run.smithi017.stdout: mon_host = [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:12:55.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1346476572' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:12:55.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:55 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1346476572' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.810 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-06T02:12:55.810 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:55.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T02:12:55.839 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:55.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:55.905 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:12:55.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:12:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:55 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1346476572' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:55.938 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:12:55.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:12:56.009 INFO:tasks.cephadm:Adding mgr.y on smithi017 2024-04-06T02:12:56.010 INFO:tasks.cephadm:Adding mgr.x on smithi104 2024-04-06T02:12:56.010 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply mgr '2;smithi017=y;smithi104=x' 2024-04-06T02:12:56.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:56 smithi104 ceph-mon[25739]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:56.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:56 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:56.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:56 smithi017 ceph-mon[29567]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:56.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:56 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:56.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:56 smithi017 ceph-mon[24844]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:56.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:56 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:57.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:12:57.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:57 smithi104 ceph-mon[25739]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:12:57.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[24844]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:12:57.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:57 smithi017 ceph-mon[29567]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:12:58.642 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mgr update... 2024-04-06T02:12:59.197 DEBUG:teuthology.orchestra.run.smithi104:mgr.x> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service 2024-04-06T02:12:59.200 INFO:tasks.cephadm:Deploying OSDs... 2024-04-06T02:12:59.200 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:12:59.200 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T02:12:59.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T02:12:59.219 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-04-06T02:12:59.276 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-04-06 02:12:35.239458625 +0000 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-04-06 02:10:11.683861647 +0000 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-04-06 02:10:11.683861647 +0000 2024-04-06T02:12:59.277 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-04-06 02:10:11.683861647 +0000 2024-04-06T02:12:59.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T02:12:59.346 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-04-06T02:12:59.346 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-04-06T02:12:59.346 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000119459 s, 4.3 MB/s 2024-04-06T02:12:59.347 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T02:12:59.403 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-04-06T02:12:59.459 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T02:12:59.459 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:12:59.459 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-06T02:12:59.459 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:12:59.459 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:12:59.460 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-04-06 02:12:35.240458615 +0000 2024-04-06T02:12:59.460 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-04-06 02:10:12.027858041 +0000 2024-04-06T02:12:59.460 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-04-06 02:10:12.027858041 +0000 2024-04-06T02:12:59.460 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-04-06 02:10:12.027858041 +0000 2024-04-06T02:12:59.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T02:12:59.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: Saving service mgr spec with placement smithi017=y;smithi104=x;count:2 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:12:59.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T02:12:59.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:12:59.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:12:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: Saving service mgr spec with placement smithi017=y;smithi104=x;count:2 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:12:59.524 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: Saving service mgr spec with placement smithi017=y;smithi104=x;count:2 2024-04-06T02:12:59.525 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:12:59.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:12:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:12:59.527 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-04-06T02:12:59.527 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-04-06T02:12:59.527 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000104477 s, 4.9 MB/s 2024-04-06T02:12:59.527 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T02:12:59.583 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:12:59.638 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-04-06 02:12:35.240458615 +0000 2024-04-06T02:12:59.639 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-04-06 02:10:12.365854498 +0000 2024-04-06T02:12:59.639 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-04-06 02:10:12.365854498 +0000 2024-04-06T02:12:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-04-06 02:10:12.365854498 +0000 2024-04-06T02:12:59.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T02:12:59.704 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-04-06T02:12:59.704 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-04-06T02:12:59.704 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.3482e-05 s, 5.5 MB/s 2024-04-06T02:12:59.705 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T02:12:59.761 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-04-06 02:12:35.241458604 +0000 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-04-06 02:10:12.695851039 +0000 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-04-06 02:10:12.695851039 +0000 2024-04-06T02:12:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-04-06 02:10:12.695851039 +0000 2024-04-06T02:12:59.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T02:12:59.881 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-04-06T02:12:59.881 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-04-06T02:12:59.881 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000104168 s, 4.9 MB/s 2024-04-06T02:12:59.883 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T02:12:59.939 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:12:59.939 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T02:12:59.954 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T02:12:59.954 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-04-06T02:13:00.008 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-06 02:12:44.632715001 +0000 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-06 02:10:08.414936361 +0000 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-06 02:10:08.414936361 +0000 2024-04-06T02:13:00.009 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-06 02:10:08.414936361 +0000 2024-04-06T02:13:00.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T02:13:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-06T02:13:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-06T02:13:00.075 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.00026422 s, 1.9 MB/s 2024-04-06T02:13:00.075 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T02:13:00.131 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-04-06T02:13:00.185 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-06 02:12:44.632715001 +0000 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-06 02:10:08.781932019 +0000 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-06 02:10:08.781932019 +0000 2024-04-06T02:13:00.186 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-06 02:10:08.781932019 +0000 2024-04-06T02:13:00.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T02:13:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-06T02:13:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-06T02:13:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000214746 s, 2.4 MB/s 2024-04-06T02:13:00.352 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T02:13:00.367 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-06 02:13:00.322527150 +0000 2024-04-06T02:13:00.427 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-06 02:10:09.107928162 +0000 2024-04-06T02:13:00.428 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-06 02:10:09.107928162 +0000 2024-04-06T02:13:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-06 02:10:09.107928162 +0000 2024-04-06T02:13:00.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T02:13:00.526 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-06T02:13:00.526 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-06T02:13:00.526 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000180749 s, 2.8 MB/s 2024-04-06T02:13:00.528 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T02:13:00.543 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-06 02:13:00.497525055 +0000 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-06 02:10:09.452924080 +0000 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-06 02:10:09.452924080 +0000 2024-04-06T02:13:00.600 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-06 02:10:09.452924080 +0000 2024-04-06T02:13:00.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T02:13:00.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:00 smithi104 ceph-mon[25739]: Deploying daemon mgr.x on smithi104 2024-04-06T02:13:00.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:00 smithi104 ceph-mon[25739]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:00.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 systemd[1]: Starting Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:13:00.668 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-06T02:13:00.668 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-06T02:13:00.668 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.00017388 s, 2.9 MB/s 2024-04-06T02:13:00.670 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T02:13:00.726 INFO:tasks.cephadm:Deploying osd.0 on smithi017 with /dev/vg_nvme/lv_4... 2024-04-06T02:13:00.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T02:13:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:00 smithi017 ceph-mon[24844]: Deploying daemon mgr.x on smithi104 2024-04-06T02:13:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:00 smithi017 ceph-mon[24844]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:00 smithi017 ceph-mon[29567]: Deploying daemon mgr.x on smithi104 2024-04-06T02:13:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:00 smithi017 ceph-mon[29567]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:00.944 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 podman[26866]: 2024-04-06 02:13:00.67351266 +0000 UTC m=+0.018805735 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-06T02:13:00.944 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 podman[26866]: 2024-04-06 02:13:00.778830581 +0000 UTC m=+0.124123648 container create 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.buildah.version=1.19.8, release=754, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-06T02:13:00.944 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 podman[26866]: 2024-04-06 02:13:00.956082184 +0000 UTC m=+0.301375253 container init 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, com.redhat.component=centos-stream-container, architecture=x86_64, vendor=Red Hat, Inc., RELEASE=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T02:13:01.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 podman[26866]: 2024-04-06 02:13:00.9587185 +0000 UTC m=+0.304011569 container start 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, io.openshift.tags=base centos centos-stream, name=centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T02:13:01.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:00 smithi104 ceph-mgr[26922]: -- 172.21.15.104:0/1143361644 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5625b4e1e340 con 0x5625b4df6800 2024-04-06T02:13:01.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:01 smithi104 bash[26866]: 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 2024-04-06T02:13:01.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:01 smithi104 systemd[1]: Started Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:13:01.827 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:01 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:01.473+0000 7fbd676e5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:13:01.827 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:01 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:01.566+0000 7fbd676e5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:13:02.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:02.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:02.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:02.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:02.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[24844]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:02.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:02.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:02.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:02.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:02.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:02 smithi017 ceph-mon[29567]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-mon[25739]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.180+0000 7fbd676e5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.268+0000 7fbd676e5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:13:02.416 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.349+0000 7fbd676e5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00801608 s, 1.3 GB/s 2024-04-06T02:13:02.730 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-04-06T02:13:02.751 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.497+0000 7fbd676e5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:13:02.752 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.576+0000 7fbd676e5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:13:02.752 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.681+0000 7fbd676e5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:13:02.752 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.763+0000 7fbd676e5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:13:03.153 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:02.924+0000 7fbd676e5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:13:03.153 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:03.008+0000 7fbd676e5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: Reconfiguring mgr.y (unknown last config time)... 2024-04-06T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:13:03.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:03.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:03 smithi104 ceph-mon[25739]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: Reconfiguring mgr.y (unknown last config time)... 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:03.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[29567]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: Reconfiguring mgr.y (unknown last config time)... 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:03 smithi017 ceph-mon[24844]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:13:04.120 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi017:vg_nvme/lv_4 2024-04-06T02:13:04.332 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.029+0000 7fbd676e5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:13:04.332 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.155+0000 7fbd676e5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:13:04.332 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.344+0000 7fbd676e5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:13:04.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.429+0000 7fbd676e5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:13:04.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.563+0000 7fbd676e5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:04.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:04.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:04.732 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:04.732 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:04.732 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:05.165 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:04.877+0000 7fbd676e5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:13:05.165 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:05.093+0000 7fbd676e5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:13:05.165 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:05.176+0000 7fbd676e5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:13:05.415 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:05.343+0000 7fbd676e5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:13:06.518 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:06.243+0000 7fbd676e5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:13:06.518 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:06.530+0000 7fbd676e5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:06.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:06.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:06.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:06.624+0000 7fbd676e5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:13:06.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:13:06.771+0000 7fbd676e5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:13:06.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:06.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:13:06.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:06.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:13:06.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:06.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:06.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: Standby manager daemon x started 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:13:07.535 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:13:07.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:07 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:13:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:07 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/237996221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:13:08.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:08.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-06T02:13:08.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:13:08.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]: dispatch 2024-04-06T02:13:08.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]': finished 2024-04-06T02:13:08.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: osdmap e5: 1 total, 0 up, 1 in 2024-04-06T02:13:08.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:08 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]: dispatch 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]': finished 2024-04-06T02:13:08.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: osdmap e5: 1 total, 0 up, 1 in 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]: dispatch 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1001984718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72fc82a0-60b7-44f9-a000-ef711644aacb"}]': finished 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: osdmap e5: 1 total, 0 up, 1 in 2024-04-06T02:13:08.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:08 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:09.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:09 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3299979926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:09.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:09 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3299979926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:09.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:09 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3299979926' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:10.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:10 smithi104 ceph-mon[25739]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:10.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:10 smithi017 ceph-mon[24844]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:10.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:10 smithi017 ceph-mon[29567]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:12 smithi017 ceph-mon[24844]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:12 smithi017 ceph-mon[29567]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:12 smithi104 ceph-mon[25739]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[24844]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[29567]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:13:14.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:14 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:14.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:14 smithi104 ceph-mon[25739]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:14.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:14 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:13:14.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:14 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:15.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:15 smithi104 ceph-mon[25739]: Deploying daemon osd.0 on smithi017 2024-04-06T02:13:15.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:15 smithi017 ceph-mon[24844]: Deploying daemon osd.0 on smithi017 2024-04-06T02:13:15.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:15 smithi017 ceph-mon[29567]: Deploying daemon osd.0 on smithi017 2024-04-06T02:13:16.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:16 smithi017 ceph-mon[24844]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:16.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:16 smithi017 ceph-mon[29567]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:16.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:16 smithi104 ceph-mon[25739]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:18.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:18 smithi017 ceph-mon[24844]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:18.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:18 smithi017 ceph-mon[29567]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:18.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:18 smithi104 ceph-mon[25739]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[24844]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:20.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[29567]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:20.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:20.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:20.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:20.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:20.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:20 smithi104 ceph-mon[25739]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:20.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:20.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:20.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:20.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:22.723 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[29567]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[29567]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[24844]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[24844]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:13:22.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:22 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:22.752 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 0 on host 'smithi017' 2024-04-06T02:13:22.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:22 smithi104 ceph-mon[25739]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:22.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:22 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:22.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:22 smithi104 ceph-mon[25739]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:13:22.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:22 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:23.462 DEBUG:teuthology.orchestra.run.smithi017:osd.0> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-06T02:13:23.464 INFO:tasks.cephadm:Deploying osd.1 on smithi017 with /dev/vg_nvme/lv_3... 2024-04-06T02:13:23.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:23.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[24844]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:23.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-mon[29567]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:23.994 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[32424]: 2024-04-06T02:13:23.937+0000 7feb1599c700 -1 osd.0 0 waiting for initial osdmap 2024-04-06T02:13:23.994 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:13:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[32424]: 2024-04-06T02:13:23.939+0000 7feb10b34700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:13:24.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T02:13:24.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:24.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:24.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:23 smithi104 ceph-mon[25739]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[24844]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[24844]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[29567]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[29567]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.121 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:24 smithi104 ceph-mon[25739]: from='osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:25.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:24 smithi104 ceph-mon[25739]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T02:13:25.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563] boot 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: osdmap e8: 1 total, 1 up, 1 in 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[29567]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:13:25.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:13:25.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563] boot 2024-04-06T02:13:25.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: osdmap e8: 1 total, 1 up, 1 in 2024-04-06T02:13:25.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:25.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:25 smithi017 ceph-mon[24844]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: osd.0 [v2:172.21.15.17:6802/669246563,v1:172.21.15.17:6803/669246563] boot 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: osdmap e8: 1 total, 1 up, 1 in 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:25 smithi104 ceph-mon[25739]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00865629 s, 1.2 GB/s 2024-04-06T02:13:26.969 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-04-06T02:13:27.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:26 smithi017 ceph-mon[24844]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T02:13:27.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:26 smithi017 ceph-mon[29567]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T02:13:27.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:26 smithi104 ceph-mon[25739]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T02:13:27.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi017:vg_nvme/lv_3 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi017 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi017 to 16069M 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[29567]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi017 2024-04-06T02:13:28.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:28.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi017 to 16069M 2024-04-06T02:13:28.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:28 smithi017 ceph-mon[24844]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi017 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi017 to 16069M 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:28.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:28 smithi104 ceph-mon[25739]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:30.726 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[29567]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:30.726 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:30.726 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:30.726 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:30.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[24844]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:30.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:30.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:30.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:30 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:30.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:30 smithi104 ceph-mon[25739]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:30.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:30 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:30.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:30 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:30.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:30 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:31.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:31 smithi017 ceph-mon[24844]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:31.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:31 smithi017 ceph-mon[29567]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:31.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:31 smithi104 ceph-mon[25739]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:32.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:32 smithi104 ceph-mon[25739]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:32.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:32 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]: dispatch 2024-04-06T02:13:32.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:32 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]': finished 2024-04-06T02:13:32.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:32 smithi104 ceph-mon[25739]: osdmap e10: 2 total, 1 up, 2 in 2024-04-06T02:13:32.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:32 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[29567]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]: dispatch 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]': finished 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[29567]: osdmap e10: 2 total, 1 up, 2 in 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:32.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[24844]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]: dispatch 2024-04-06T02:13:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4072668258' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ca29e90-938b-4c53-896d-20661c1cd1d6"}]': finished 2024-04-06T02:13:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[24844]: osdmap e10: 2 total, 1 up, 2 in 2024-04-06T02:13:32.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:32 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:33.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:33 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/375989126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:33.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:33 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/375989126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:33.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:33 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/375989126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:34.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:34 smithi017 ceph-mon[24844]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:34.849 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:34 smithi017 ceph-mon[29567]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:34 smithi104 ceph-mon[25739]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:36.882 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:36 smithi017 ceph-mon[24844]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:36.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:36 smithi017 ceph-mon[29567]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:36.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:36 smithi104 ceph-mon[25739]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:38.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[29567]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:38.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:13:38.839 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:38.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[24844]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:38.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:13:38.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:38.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:38 smithi104 ceph-mon[25739]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:38.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:13:38.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:39.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:39 smithi017 ceph-mon[24844]: Deploying daemon osd.1 on smithi017 2024-04-06T02:13:39.718 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:39 smithi017 ceph-mon[29567]: Deploying daemon osd.1 on smithi017 2024-04-06T02:13:39.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:39 smithi104 ceph-mon[25739]: Deploying daemon osd.1 on smithi017 2024-04-06T02:13:40.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:40 smithi104 ceph-mon[25739]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:40.969 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:40 smithi017 ceph-mon[24844]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:40.970 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:40 smithi017 ceph-mon[29567]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:42.599 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:42 smithi017 ceph-mon[29567]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:42.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:42 smithi017 ceph-mon[24844]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:42 smithi104 ceph-mon[25739]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:44 smithi104 ceph-mon[25739]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[24844]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:44.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:44 smithi017 ceph-mon[29567]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:46.259 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 1 on host 'smithi017' 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[24844]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[24844]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[29567]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[29567]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:13:46.841 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:46 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:46.867 DEBUG:teuthology.orchestra.run.smithi017:osd.1> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-06T02:13:46.870 INFO:tasks.cephadm:Deploying osd.2 on smithi017 with /dev/vg_nvme/lv_2... 2024-04-06T02:13:46.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T02:13:46.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:46 smithi104 ceph-mon[25739]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:46.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:46 smithi104 ceph-mon[25739]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:13:46.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:46 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:47.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[24844]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[24844]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[24844]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[29567]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[29567]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[29567]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:47.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:47.743 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[35392]: 2024-04-06T02:13:47.560+0000 7ff0b848e700 -1 osd.1 0 waiting for initial osdmap 2024-04-06T02:13:47.743 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:13:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[35392]: 2024-04-06T02:13:47.562+0000 7ff0b0e21700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:13:47.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:47 smithi104 ceph-mon[25739]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:13:47.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:47 smithi104 ceph-mon[25739]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T02:13:47.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:47 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:47.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:47 smithi104 ceph-mon[25739]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:13:47.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:47 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:48.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[24844]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:48.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[24844]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:48.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[24844]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T02:13:48.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:48.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:48.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[29567]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:48.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[29567]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:48.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[29567]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T02:13:48.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:48.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:48 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:48.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:48 smithi104 ceph-mon[25739]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:48.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:48 smithi104 ceph-mon[25739]: from='osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:13:48.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:48 smithi104 ceph-mon[25739]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T02:13:48.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:48 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:48.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:48 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:49.535 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T02:13:49.536 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T02:13:49.536 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-04-06T02:13:49.536 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-04-06T02:13:49.536 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0100483 s, 1.0 GB/s 2024-04-06T02:13:49.536 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[24844]: osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571] boot 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[24844]: osdmap e13: 2 total, 2 up, 2 in 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[29567]: osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571] boot 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[29567]: osdmap e13: 2 total, 2 up, 2 in 2024-04-06T02:13:49.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:49.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:49 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:13:49.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:49 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:13:49.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:49 smithi104 ceph-mon[25739]: osd.1 [v2:172.21.15.17:6810/320210571,v1:172.21.15.17:6811/320210571] boot 2024-04-06T02:13:49.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:49 smithi104 ceph-mon[25739]: osdmap e13: 2 total, 2 up, 2 in 2024-04-06T02:13:49.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:13:50.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi017:vg_nvme/lv_2 2024-04-06T02:13:50.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:50 smithi017 ceph-mon[24844]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:50.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:50 smithi017 ceph-mon[24844]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T02:13:50.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:50 smithi017 ceph-mon[29567]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:50.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:50 smithi017 ceph-mon[29567]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T02:13:50.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:50 smithi104 ceph-mon[25739]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:13:50.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:50 smithi104 ceph-mon[25739]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T02:13:52.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:52.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:52.583 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:52.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:13:52.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:52.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:13:52.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:13:53.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi017 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi017 to 8034M 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi017 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi017 to 8034M 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:53.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:53 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:53.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:53 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi017 2024-04-06T02:13:53.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:53 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi017 to 8034M 2024-04-06T02:13:53.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:53 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:13:53.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:53 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:13:53.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:53 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:13:54.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:54 smithi104 ceph-mon[25739]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:54.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:54 smithi104 ceph-mon[25739]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:54.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:54 smithi017 ceph-mon[24844]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:54.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:54 smithi017 ceph-mon[24844]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:54.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:54 smithi017 ceph-mon[29567]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:13:54.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:54 smithi017 ceph-mon[29567]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:55 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1643040936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:55 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:55 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]': finished 2024-04-06T02:13:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:55 smithi104 ceph-mon[25739]: osdmap e15: 3 total, 2 up, 3 in 2024-04-06T02:13:55.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:55 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1643040936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]': finished 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[24844]: osdmap e15: 3 total, 2 up, 3 in 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1643040936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]: dispatch 2024-04-06T02:13:55.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bb8c263-c05c-488e-9574-310742483a34"}]': finished 2024-04-06T02:13:55.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[29567]: osdmap e15: 3 total, 2 up, 3 in 2024-04-06T02:13:55.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:55 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:13:56.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:56 smithi104 ceph-mon[25739]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:56.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:56 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1443009151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:56.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:56 smithi017 ceph-mon[29567]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:56.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:56 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1443009151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:56 smithi017 ceph-mon[24844]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:56.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:56 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1443009151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:13:58.826 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:13:58 smithi017 ceph-mon[24844]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:58.827 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:13:58 smithi017 ceph-mon[29567]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:13:58.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:13:58 smithi104 ceph-mon[25739]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:00.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:00 smithi017 ceph-mon[24844]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:00 smithi017 ceph-mon[29567]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:00.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:00 smithi104 ceph-mon[25739]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:02.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[24844]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:02.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:14:02.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:02.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[24844]: Deploying daemon osd.2 on smithi017 2024-04-06T02:14:02.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[29567]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:02.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:14:02.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:02.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:02 smithi017 ceph-mon[29567]: Deploying daemon osd.2 on smithi017 2024-04-06T02:14:02.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:02 smithi104 ceph-mon[25739]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:02.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:14:02.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:02 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:02.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:02 smithi104 ceph-mon[25739]: Deploying daemon osd.2 on smithi017 2024-04-06T02:14:04.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:04 smithi017 ceph-mon[24844]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:04.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:04 smithi017 ceph-mon[29567]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:04.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:04 smithi104 ceph-mon[25739]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:05.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:14:05.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:14:05.831 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:05 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:14:05.831 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:05 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:14:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:05 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:14:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:05 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:14:06.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:06 smithi017 ceph-mon[24844]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:06 smithi017 ceph-mon[29567]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:06.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:06 smithi104 ceph-mon[25739]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:07.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:07.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[29567]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:07.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:07 smithi017 ceph-mon[24844]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:08.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:07 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:08.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:07 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:08.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:07 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:08.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:07 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:08.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:07 smithi104 ceph-mon[25739]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:09.758 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 2 on host 'smithi017' 2024-04-06T02:14:10.372 DEBUG:teuthology.orchestra.run.smithi017:osd.2> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-06T02:14:10.375 INFO:tasks.cephadm:Deploying osd.3 on smithi017 with /dev/vg_nvme/lv_1... 2024-04-06T02:14:10.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[24844]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[24844]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[29567]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:10.627 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:10 smithi017 ceph-mon[29567]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:14:10.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:10 smithi104 ceph-mon[25739]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:10.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:10 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:10.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:10 smithi104 ceph-mon[25739]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[29567]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[29567]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[29567]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[24844]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[24844]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T02:14:11.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:11.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-mon[24844]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:11.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:11 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:11.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:11 smithi104 ceph-mon[25739]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:14:11.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:11 smithi104 ceph-mon[25739]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T02:14:11.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:11 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:11.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:11 smithi104 ceph-mon[25739]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:11.991 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[38326]: 2024-04-06T02:14:11.775+0000 7fdfb7ee1700 -1 osd.2 0 waiting for initial osdmap 2024-04-06T02:14:11.991 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:11 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[38326]: 2024-04-06T02:14:11.779+0000 7fdfb2878700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:14:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[24844]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[24844]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[24844]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[29567]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[29567]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[29567]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:12 smithi104 ceph-mon[25739]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:12 smithi104 ceph-mon[25739]: from='osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:12 smithi104 ceph-mon[25739]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T02:14:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010262 s, 1.0 GB/s 2024-04-06T02:14:13.080 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-04-06T02:14:13.807 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[29567]: osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685] boot 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[29567]: osdmap e18: 3 total, 3 up, 3 in 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[24844]: osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685] boot 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[24844]: osdmap e18: 3 total, 3 up, 3 in 2024-04-06T02:14:13.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:13 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:13.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi017:vg_nvme/lv_1 2024-04-06T02:14:14.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:13 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:14:14.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:13 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:14:14.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:13 smithi104 ceph-mon[25739]: osd.2 [v2:172.21.15.17:6818/1751363685,v1:172.21.15.17:6819/1751363685] boot 2024-04-06T02:14:14.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:13 smithi104 ceph-mon[25739]: osdmap e18: 3 total, 3 up, 3 in 2024-04-06T02:14:14.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:13 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[29567]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[29567]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[24844]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[24844]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T02:14:14.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:14 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:14:15.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:14 smithi104 ceph-mon[25739]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:14:15.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:14 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-06T02:14:15.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:14 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-06T02:14:15.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:14 smithi104 ceph-mon[25739]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T02:14:15.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:14 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi017 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi017 to 5356M 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:16.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:15 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.078 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:15 smithi017 sudo[39614]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:16.078 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:15 smithi017 sudo[39614]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:16.078 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:15 smithi017 sudo[39614]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:16.078 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi017 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi017 to 5356M 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:16.079 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi017 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi017 to 5356M 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:16.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:16.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:16.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:15 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39617]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:16.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39617]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:16.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39617]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:16.622 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39638]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:16.622 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39638]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:16.622 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39638]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:16.879 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39667]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:16.879 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39667]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:16.879 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39667]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:16.879 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[29567]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T02:14:16.879 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.879 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:16.879 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[24844]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:16.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:16.880 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39707]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:17.150 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39707]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:17.151 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:16 smithi017 sudo[39707]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:17.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:16 smithi104 ceph-mon[25739]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T02:14:17.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:17.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:17.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:17.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:17.403 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39753]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:17.403 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39753]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:17.403 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39753]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:17.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39762]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:17.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39762]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:17.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39762]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:17.935 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39777]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:17.935 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39777]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:17.935 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39777]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39783]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39783]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:17 smithi017 sudo[39783]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:18.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 sudo[39805]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:18.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 sudo[39805]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:18.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 sudo[39805]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:18.699 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27249]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:14:18.699 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27249]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:18.699 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27249]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:18.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:18.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27252]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27252]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 sudo[27252]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:19.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:19.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:19.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:19.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:19.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:19.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]: dispatch 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]': finished 2024-04-06T02:14:19.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: osdmap e22: 4 total, 3 up, 4 in 2024-04-06T02:14:19.994 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:19.994 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[29567]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:19.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:19.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:19.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]: dispatch 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]': finished 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: osdmap e22: 4 total, 3 up, 4 in 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:19.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:19 smithi017 ceph-mon[24844]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-06T02:14:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:20.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]: dispatch 2024-04-06T02:14:20.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4231294259' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92befe7c-2f6f-4c88-a41b-58cc7dd04c46"}]': finished 2024-04-06T02:14:20.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: osdmap e22: 4 total, 3 up, 4 in 2024-04-06T02:14:20.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:20.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:19 smithi104 ceph-mon[25739]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:20.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:20 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3804291427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:20.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:20 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3804291427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:21.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:20 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3804291427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:21.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:21 smithi017 ceph-mon[24844]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:21.973 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:21 smithi017 ceph-mon[29567]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:22.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:21 smithi104 ceph-mon[25739]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:24.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:24 smithi017 ceph-mon[24844]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:24.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:24 smithi017 ceph-mon[29567]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:24.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:24 smithi104 ceph-mon[25739]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:26 smithi104 ceph-mon[25739]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:26 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:14:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:26 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:26 smithi104 ceph-mon[25739]: Deploying daemon osd.3 on smithi017 2024-04-06T02:14:26.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[24844]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:26.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:14:26.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:26.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[24844]: Deploying daemon osd.3 on smithi017 2024-04-06T02:14:26.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[29567]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:26.918 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:14:26.918 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:26.918 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:26 smithi017 ceph-mon[29567]: Deploying daemon osd.3 on smithi017 2024-04-06T02:14:28.809 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:28 smithi017 ceph-mon[24844]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:28.809 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:28 smithi017 ceph-mon[29567]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:28.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:28 smithi104 ceph-mon[25739]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:30.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:30 smithi017 ceph-mon[24844]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:30.659 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:30 smithi017 ceph-mon[29567]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:30.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:30 smithi104 ceph-mon[25739]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:31.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:31.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:31.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:31.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:31.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[24844]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:31.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:31.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:31.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:31.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:31.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:31 smithi017 ceph-mon[29567]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:32.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:31 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:32.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:31 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:32.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:31 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:32.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:31 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:32.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:31 smithi104 ceph-mon[25739]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:34.315 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 3 on host 'smithi017' 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:34.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:34 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:14:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:34 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:34.977 DEBUG:teuthology.orchestra.run.smithi017:osd.3> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-06T02:14:34.979 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_4... 2024-04-06T02:14:34.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T02:14:35.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:35 smithi104 ceph-mon[25739]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:14:35.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:35 smithi104 ceph-mon[25739]: osdmap e23: 4 total, 3 up, 4 in 2024-04-06T02:14:35.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:35 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:35.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:35 smithi104 ceph-mon[25739]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:35.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[24844]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[24844]: osdmap e23: 4 total, 3 up, 4 in 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[24844]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[29567]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[29567]: osdmap e23: 4 total, 3 up, 4 in 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:35.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-mon[29567]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:14:35.992 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[41309]: 2024-04-06T02:14:35.570+0000 7f3e455bc700 -1 osd.3 0 waiting for initial osdmap 2024-04-06T02:14:35.993 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:14:35 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[41309]: 2024-04-06T02:14:35.575+0000 7f3e3ef51700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:14:36.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:36 smithi104 ceph-mon[25739]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:36.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:36 smithi104 ceph-mon[25739]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:36.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:36 smithi104 ceph-mon[25739]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T02:14:36.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[24844]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[24844]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[24844]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[29567]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[29567]: from='osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[29567]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T02:14:36.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:36 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:36.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:36 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115055 s, 911 MB/s 2024-04-06T02:14:37.027 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-06T02:14:37.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[29567]: osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733] boot 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[29567]: osdmap e25: 4 total, 4 up, 4 in 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:37.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:14:37.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:14:37.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[24844]: osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733] boot 2024-04-06T02:14:37.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[24844]: osdmap e25: 4 total, 4 up, 4 in 2024-04-06T02:14:37.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:37 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:37.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:37 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:14:37.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:37 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:14:37.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:37 smithi104 ceph-mon[25739]: osd.3 [v2:172.21.15.17:6826/2170379733,v1:172.21.15.17:6827/2170379733] boot 2024-04-06T02:14:37.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:37 smithi104 ceph-mon[25739]: osdmap e25: 4 total, 4 up, 4 in 2024-04-06T02:14:37.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:37 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: osdmap e26: 4 total, 4 up, 4 in 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:38 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:38.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: osdmap e26: 4 total, 4 up, 4 in 2024-04-06T02:14:38.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: osdmap e26: 4 total, 4 up, 4 in 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:14:38.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:38.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:38 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:39.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:39 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi017 2024-04-06T02:14:39.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:39 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi017 to 4017M 2024-04-06T02:14:39.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:39 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:39.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:39 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:39.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:39 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:39.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi017 2024-04-06T02:14:39.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi017 to 4017M 2024-04-06T02:14:39.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:39.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:39.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:39.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi017 2024-04-06T02:14:39.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi017 to 4017M 2024-04-06T02:14:39.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:14:39.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:14:39.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:39 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:40 smithi104 ceph-mon[25739]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:40 smithi104 ceph-mon[25739]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:40.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:40 smithi017 ceph-mon[29567]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:40.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:40 smithi017 ceph-mon[29567]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:40.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:40 smithi017 ceph-mon[24844]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:40.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:40 smithi017 ceph-mon[24844]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/752910003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]': finished 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: osdmap e27: 5 total, 4 up, 5 in 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:42.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/2637316685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/752910003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]': finished 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: osdmap e27: 5 total, 4 up, 5 in 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/2637316685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/752910003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]: dispatch 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8b86c23-048b-4913-bc79-b75e88d8425b"}]': finished 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: osdmap e27: 5 total, 4 up, 5 in 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:42.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/2637316685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:14:44.760 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:44 smithi104 ceph-mon[25739]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:44 smithi017 ceph-mon[24844]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:44.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:44 smithi017 ceph-mon[29567]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:46.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:46 smithi104 ceph-mon[25739]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:46.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:46 smithi017 ceph-mon[24844]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:46.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:46 smithi017 ceph-mon[29567]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:48.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:48 smithi104 ceph-mon[25739]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:48.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:48 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:14:48.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:48 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:48.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:48 smithi104 ceph-mon[25739]: Deploying daemon osd.4 on smithi104 2024-04-06T02:14:48.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[24844]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:48.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:14:48.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:48.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[24844]: Deploying daemon osd.4 on smithi104 2024-04-06T02:14:48.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[29567]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:48.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:14:48.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:48.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:48 smithi017 ceph-mon[29567]: Deploying daemon osd.4 on smithi104 2024-04-06T02:14:50.741 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:50 smithi104 ceph-mon[25739]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:50 smithi017 ceph-mon[24844]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:50.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:50 smithi017 ceph-mon[29567]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:52 smithi104 ceph-mon[25739]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:52 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:52.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[24844]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[29567]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:14:52.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:52 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:14:54.628 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:54 smithi104 ceph-mon[25739]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:54.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:54 smithi017 ceph-mon[24844]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:54.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:54 smithi017 ceph-mon[29567]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:55.470 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-04-06T02:14:56.074 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-06T02:14:56.076 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_3... 2024-04-06T02:14:56.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T02:14:56.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:56 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:56.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:56 smithi104 ceph-mon[25739]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:56.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:56 smithi104 ceph-mon[25739]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:56 smithi104 ceph-mon[25739]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:56 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:56.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:56.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[24844]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:56.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[24844]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[24844]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:56.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:56.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[29567]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:56.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[29567]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[29567]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:14:56.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:56 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:14:57.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[24844]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:14:57.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[24844]: osdmap e28: 5 total, 4 up, 5 in 2024-04-06T02:14:57.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:57.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[24844]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[24844]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[29567]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:14:57.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[29567]: osdmap e28: 5 total, 4 up, 5 in 2024-04-06T02:14:57.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:57.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[29567]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:57 smithi017 ceph-mon[29567]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-mon[25739]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:14:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-mon[25739]: osdmap e28: 5 total, 4 up, 5 in 2024-04-06T02:14:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-mon[25739]: from='osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-mon[25739]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:14:57.796 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[29089]: 2024-04-06T02:14:57.494+0000 7f862ea3a700 -1 osd.4 0 waiting for initial osdmap 2024-04-06T02:14:57.796 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:14:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[29089]: 2024-04-06T02:14:57.497+0000 7f86283cf700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:14:58.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:58 smithi104 ceph-mon[25739]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:14:58.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:58 smithi104 ceph-mon[25739]: osdmap e29: 5 total, 4 up, 5 in 2024-04-06T02:14:58.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:58 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:58 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:58 smithi104 ceph-mon[25739]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[29567]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[29567]: osdmap e29: 5 total, 4 up, 5 in 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[29567]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[24844]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:14:58.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[24844]: osdmap e29: 5 total, 4 up, 5 in 2024-04-06T02:14:58.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:58.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:58 smithi017 ceph-mon[24844]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348] boot 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: osdmap e30: 5 total, 5 up, 5 in 2024-04-06T02:14:59.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348] boot 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: osdmap e30: 5 total, 5 up, 5 in 2024-04-06T02:14:59.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:14:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: osd.4 [v2:172.21.15.104:6800/260389348,v1:172.21.15.104:6801/260389348] boot 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: osdmap e30: 5 total, 5 up, 5 in 2024-04-06T02:14:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:14:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128028 s, 819 MB/s 2024-04-06T02:15:00.240 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-06T02:15:00.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: osdmap e31: 5 total, 5 up, 5 in 2024-04-06T02:15:00.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:00.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: osdmap e31: 5 total, 5 up, 5 in 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:00.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:00.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:00.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:00.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:00 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.852 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: osdmap e31: 5 total, 5 up, 5 in 2024-04-06T02:15:00.852 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:00.853 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:00 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:00.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi104 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi104 to 17088M 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[29567]: osdmap e32: 5 total, 5 up, 5 in 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi104 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi104 to 17088M 2024-04-06T02:15:01.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:01 smithi017 ceph-mon[24844]: osdmap e32: 5 total, 5 up, 5 in 2024-04-06T02:15:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:01 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi104 2024-04-06T02:15:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:01 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi104 to 17088M 2024-04-06T02:15:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:01 smithi104 ceph-mon[25739]: osdmap e32: 5 total, 5 up, 5 in 2024-04-06T02:15:02.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:02 smithi104 ceph-mon[25739]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:02.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:02 smithi104 ceph-mon[25739]: osdmap e33: 5 total, 5 up, 5 in 2024-04-06T02:15:02.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:02 smithi017 ceph-mon[24844]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:02.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:02 smithi017 ceph-mon[24844]: osdmap e33: 5 total, 5 up, 5 in 2024-04-06T02:15:02.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:02 smithi017 ceph-mon[29567]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:02.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:02 smithi017 ceph-mon[29567]: osdmap e33: 5 total, 5 up, 5 in 2024-04-06T02:15:03.766 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:03 smithi104 ceph-mon[25739]: osdmap e34: 5 total, 5 up, 5 in 2024-04-06T02:15:03.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:03.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:03.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:03 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:03.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[24844]: osdmap e34: 5 total, 5 up, 5 in 2024-04-06T02:15:03.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:03.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:03.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:03.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[29567]: osdmap e34: 5 total, 5 up, 5 in 2024-04-06T02:15:03.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:03.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:03.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:03 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:04.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:04 smithi104 ceph-mon[25739]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:04.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:04 smithi104 ceph-mon[25739]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:04 smithi017 ceph-mon[24844]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:04 smithi017 ceph-mon[24844]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:04 smithi017 ceph-mon[29567]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:04 smithi017 ceph-mon[29567]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:05.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:05 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/1327936294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:05 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:05 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]': finished 2024-04-06T02:15:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:05 smithi104 ceph-mon[25739]: osdmap e35: 6 total, 5 up, 6 in 2024-04-06T02:15:05.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:05 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:05.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/1327936294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]': finished 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[24844]: osdmap e35: 6 total, 5 up, 6 in 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/1327936294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]: dispatch 2024-04-06T02:15:05.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff15491f-631c-40d1-afb0-0612c6bd6c60"}]': finished 2024-04-06T02:15:05.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[29567]: osdmap e35: 6 total, 5 up, 6 in 2024-04-06T02:15:05.993 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:05 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:06.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:06 smithi104 ceph-mon[25739]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:15:06.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:15:06.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:06 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:15:06.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:06 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/335117819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[24844]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/335117819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[29567]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:15:06.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:15:06.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:15:06.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:06 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/335117819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:08 smithi104 ceph-mon[25739]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:15:08.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:08 smithi017 ceph-mon[29567]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:15:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:08 smithi017 ceph-mon[24844]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:15:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:10 smithi104 ceph-mon[25739]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:10.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:10 smithi017 ceph-mon[24844]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:10.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:10 smithi017 ceph-mon[29567]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:11.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:11 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:15:11.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:11 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:11.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:11 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:15:11.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:11 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:11.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:11 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:15:11.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:11 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:12 smithi104 ceph-mon[25739]: Deploying daemon osd.5 on smithi104 2024-04-06T02:15:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:12 smithi104 ceph-mon[25739]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-06T02:15:12.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:12 smithi017 ceph-mon[24844]: Deploying daemon osd.5 on smithi104 2024-04-06T02:15:12.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:12 smithi017 ceph-mon[24844]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-06T02:15:12.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:12 smithi017 ceph-mon[29567]: Deploying daemon osd.5 on smithi104 2024-04-06T02:15:12.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:12 smithi017 ceph-mon[29567]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-06T02:15:14.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:14 smithi104 ceph-mon[25739]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:15:14.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:14 smithi017 ceph-mon[29567]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:15:14.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:14 smithi017 ceph-mon[24844]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:15:16.878 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:16 smithi104 ceph-mon[25739]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:16.878 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:16.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:16.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:16.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:16 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:16.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[29567]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:16.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:16.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:16.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:16.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:16.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[24844]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:16.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:16.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:16.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:16.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:16 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:18.852 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:18 smithi104 ceph-mon[25739]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:18 smithi017 ceph-mon[29567]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:18 smithi017 ceph-mon[24844]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:19.918 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-04-06T02:15:20.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:20.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:20.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:20 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:20.489 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:20.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:20.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:20 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:20.493 INFO:tasks.cephadm:Deploying osd.6 on smithi104 with /dev/vg_nvme/lv_2... 2024-04-06T02:15:20.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T02:15:21.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-mon[25739]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:15:21.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-mon[25739]: osdmap e36: 6 total, 5 up, 6 in 2024-04-06T02:15:21.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:21.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-mon[25739]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:21.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-mon[25739]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:21.915 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[32037]: 2024-04-06T02:15:21.526+0000 7fb7bcd89700 -1 osd.5 0 waiting for initial osdmap 2024-04-06T02:15:21.915 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:15:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[32037]: 2024-04-06T02:15:21.531+0000 7fb7b8722700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[29567]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[29567]: osdmap e36: 6 total, 5 up, 6 in 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[29567]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[29567]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:21.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[24844]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:15:21.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[24844]: osdmap e36: 6 total, 5 up, 6 in 2024-04-06T02:15:21.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:21.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[24844]: from='osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:21.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:21 smithi017 ceph-mon[24844]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:22.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:22 smithi104 ceph-mon[25739]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:22.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:22 smithi104 ceph-mon[25739]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:22.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:22 smithi104 ceph-mon[25739]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T02:15:22.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:22 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:22.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:22 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[29567]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[29567]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[29567]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[24844]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[24844]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[24844]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:22.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:22 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125321 s, 837 MB/s 2024-04-06T02:15:23.795 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-06T02:15:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:15:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088] boot 2024-04-06T02:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: osdmap e38: 6 total, 6 up, 6 in 2024-04-06T02:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:23 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088] boot 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: osdmap e38: 6 total, 6 up, 6 in 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:23.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: osd.5 [v2:172.21.15.104:6808/3760683088,v1:172.21.15.104:6809/3760683088] boot 2024-04-06T02:15:23.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: osdmap e38: 6 total, 6 up, 6 in 2024-04-06T02:15:23.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:23 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:15:24.882 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:24.882 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: osdmap e39: 6 total, 6 up, 6 in 2024-04-06T02:15:24.882 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.882 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.882 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:24 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: osdmap e39: 6 total, 6 up, 6 in 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: osdmap e39: 6 total, 6 up, 6 in 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:24.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:24 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:25 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi104 2024-04-06T02:15:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:25 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi104 to 8544M 2024-04-06T02:15:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:25 smithi104 ceph-mon[25739]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T02:15:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi104 2024-04-06T02:15:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi104 to 8544M 2024-04-06T02:15:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[24844]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T02:15:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi104 2024-04-06T02:15:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi104 to 8544M 2024-04-06T02:15:25.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:25 smithi017 ceph-mon[29567]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T02:15:26.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:26 smithi104 ceph-mon[25739]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:26.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:26 smithi104 ceph-mon[25739]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T02:15:26.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:26 smithi017 ceph-mon[24844]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:26.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:26 smithi017 ceph-mon[24844]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T02:15:26.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:26 smithi017 ceph-mon[29567]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:26.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:26 smithi017 ceph-mon[29567]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T02:15:27.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[24844]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T02:15:27.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:27.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:27.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:27.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[29567]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T02:15:27.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:27.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:27.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:27 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:27.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:27 smithi104 ceph-mon[25739]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T02:15:27.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:27 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:27.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:27 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:27.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:27 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:28.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:28 smithi017 ceph-mon[24844]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:28.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:28 smithi017 ceph-mon[24844]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:28.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:28 smithi017 ceph-mon[29567]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:28.909 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:28 smithi017 ceph-mon[29567]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:28.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:28 smithi104 ceph-mon[25739]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:28.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:28 smithi104 ceph-mon[25739]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:29.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/1198581284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]': finished 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[29567]: osdmap e43: 7 total, 6 up, 7 in 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/1198581284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]': finished 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[24844]: osdmap e43: 7 total, 6 up, 7 in 2024-04-06T02:15:29.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:29 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:29.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:29 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/1198581284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:29 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]: dispatch 2024-04-06T02:15:29.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:29 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec70d0cd-b89d-4885-b7b6-e318c10a3bd5"}]': finished 2024-04-06T02:15:29.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:29 smithi104 ceph-mon[25739]: osdmap e43: 7 total, 6 up, 7 in 2024-04-06T02:15:29.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:29 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:30.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:30 smithi104 ceph-mon[25739]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:30.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:30 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/3180746763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:30.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:30 smithi017 ceph-mon[29567]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:30.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:30 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/3180746763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:30.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:30 smithi017 ceph-mon[24844]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:30.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:30 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/3180746763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:32.639 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:32 smithi104 ceph-mon[25739]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:32.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:32 smithi017 ceph-mon[24844]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:32.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:32 smithi017 ceph-mon[29567]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:34.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:34 smithi104 ceph-mon[25739]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:34.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:34 smithi017 ceph-mon[29567]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:34.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:34 smithi017 ceph-mon[24844]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:35.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:35 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:15:35.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:35 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:35.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:35 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:15:35.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:35 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:35.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:35 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:15:35.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:35 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:36.848 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:36 smithi104 ceph-mon[25739]: Deploying daemon osd.6 on smithi104 2024-04-06T02:15:36.848 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:36 smithi104 ceph-mon[25739]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:36.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:36 smithi017 ceph-mon[29567]: Deploying daemon osd.6 on smithi104 2024-04-06T02:15:36.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:36 smithi017 ceph-mon[29567]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:36.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:36 smithi017 ceph-mon[24844]: Deploying daemon osd.6 on smithi104 2024-04-06T02:15:36.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:36 smithi017 ceph-mon[24844]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:38 smithi104 ceph-mon[25739]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:38.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:38 smithi017 ceph-mon[24844]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:38.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:38 smithi017 ceph-mon[29567]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:40.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[24844]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:40.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[29567]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:40.724 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:40 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:40.778 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:40 smithi104 ceph-mon[25739]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:40.779 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:40 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:40.779 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:40 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:40.779 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:40 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:40.779 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:40 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:42.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:42 smithi104 ceph-mon[25739]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:42.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:42 smithi017 ceph-mon[24844]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:42 smithi017 ceph-mon[29567]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:43.649 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:43 smithi104 ceph-mon[25739]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:43.649 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:43 smithi104 ceph-mon[25739]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:43.680 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 6 on host 'smithi104' 2024-04-06T02:15:43.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:43 smithi017 ceph-mon[29567]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:43.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:43 smithi017 ceph-mon[29567]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:43.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:43 smithi017 ceph-mon[24844]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:43.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:43 smithi017 ceph-mon[24844]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:15:44.305 DEBUG:teuthology.orchestra.run.smithi104:osd.6> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-06T02:15:44.308 INFO:tasks.cephadm:Deploying osd.7 on smithi104 with /dev/vg_nvme/lv_1... 2024-04-06T02:15:44.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T02:15:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: osdmap e44: 7 total, 6 up, 7 in 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:44.863 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[34920]: 2024-04-06T02:15:44.593+0000 7f3f4e7d9700 -1 osd.6 0 waiting for initial osdmap 2024-04-06T02:15:44.863 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:15:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[34920]: 2024-04-06T02:15:44.601+0000 7f3f4a973700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: osdmap e44: 7 total, 6 up, 7 in 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: osdmap e44: 7 total, 6 up, 7 in 2024-04-06T02:15:44.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:44.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:15:44.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:44.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:44 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:45.902 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:45.902 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: osdmap e45: 7 total, 6 up, 7 in 2024-04-06T02:15:45.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818] boot 2024-04-06T02:15:45.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: osdmap e46: 7 total, 7 up, 7 in 2024-04-06T02:15:45.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:45 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: osdmap e45: 7 total, 6 up, 7 in 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818] boot 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: osdmap e46: 7 total, 7 up, 7 in 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:15:45.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: osdmap e45: 7 total, 6 up, 7 in 2024-04-06T02:15:45.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:45.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: osd.6 [v2:172.21.15.104:6816/579793818,v1:172.21.15.104:6817/579793818] boot 2024-04-06T02:15:45.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: osdmap e46: 7 total, 7 up, 7 in 2024-04-06T02:15:45.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:45 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:15:46.907 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T02:15:46.908 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T02:15:46.908 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-06T02:15:46.908 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-06T02:15:46.908 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124837 s, 840 MB/s 2024-04-06T02:15:46.908 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-06T02:15:46.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:46 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:15:46.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:46 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:15:46.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:46 smithi104 ceph-mon[25739]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:46.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:46 smithi104 ceph-mon[25739]: osdmap e47: 7 total, 7 up, 7 in 2024-04-06T02:15:46.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[24844]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[24844]: osdmap e47: 7 total, 7 up, 7 in 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[29567]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:15:46.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:46 smithi017 ceph-mon[29567]: osdmap e47: 7 total, 7 up, 7 in 2024-04-06T02:15:47.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-04-06T02:15:48.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:48 smithi104 ceph-mon[25739]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:48 smithi104 ceph-mon[25739]: osdmap e48: 7 total, 7 up, 7 in 2024-04-06T02:15:48.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:48 smithi017 ceph-mon[24844]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:48.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:48 smithi017 ceph-mon[24844]: osdmap e48: 7 total, 7 up, 7 in 2024-04-06T02:15:48.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:48 smithi017 ceph-mon[29567]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:48.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:48 smithi017 ceph-mon[29567]: osdmap e48: 7 total, 7 up, 7 in 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: osdmap e49: 7 total, 7 up, 7 in 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:49.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:49.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:49.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:49 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: osdmap e49: 7 total, 7 up, 7 in 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:49.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: osdmap e49: 7 total, 7 up, 7 in 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:15:49.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:49 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:15:50.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:50 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi104 2024-04-06T02:15:50.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:50 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi104 to 5696M 2024-04-06T02:15:50.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:50 smithi104 ceph-mon[25739]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:50.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:50 smithi104 ceph-mon[25739]: osdmap e50: 7 total, 7 up, 7 in 2024-04-06T02:15:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi104 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi104 to 5696M 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[29567]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[29567]: osdmap e50: 7 total, 7 up, 7 in 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi104 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi104 to 5696M 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[24844]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:15:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:50 smithi017 ceph-mon[24844]: osdmap e50: 7 total, 7 up, 7 in 2024-04-06T02:15:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:51 smithi104 ceph-mon[25739]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:51 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:51 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:51 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:51.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[29567]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[24844]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:15:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:51 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:52.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:52 smithi104 ceph-mon[25739]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T02:15:52.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:52 smithi017 ceph-mon[29567]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T02:15:52.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:52 smithi017 ceph-mon[24844]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/3110511508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]': finished 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: osdmap e51: 8 total, 7 up, 8 in 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:15:53.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:53 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/2428371707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:53.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/3110511508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]': finished 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: osdmap e51: 8 total, 7 up, 8 in 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/2428371707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/3110511508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4e3423d-e4c3-448d-9e21-c12c85405720"}]': finished 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: osdmap e51: 8 total, 7 up, 8 in 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:15:53.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:53 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/2428371707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:15:54.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:54 smithi104 ceph-mon[25739]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-06T02:15:54.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:54 smithi017 ceph-mon[24844]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-06T02:15:54.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:54 smithi017 ceph-mon[29567]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-06T02:15:56.852 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:56 smithi104 ceph-mon[25739]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-06T02:15:56.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:56 smithi017 ceph-mon[29567]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-06T02:15:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:56 smithi017 ceph-mon[24844]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-06T02:15:58.779 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:58 smithi104 ceph-mon[25739]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:58.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:58 smithi017 ceph-mon[24844]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:58.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:58 smithi017 ceph-mon[29567]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-06T02:15:59.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:15:59.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:15:59 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:59.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:15:59.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:15:59 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:15:59.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:15:59.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:15:59 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:00.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:00 smithi104 ceph-mon[25739]: Deploying daemon osd.7 on smithi104 2024-04-06T02:16:00.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:00 smithi104 ceph-mon[25739]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:16:00.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:00 smithi017 ceph-mon[24844]: Deploying daemon osd.7 on smithi104 2024-04-06T02:16:00.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:00 smithi017 ceph-mon[24844]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:16:00.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:00 smithi017 ceph-mon[29567]: Deploying daemon osd.7 on smithi104 2024-04-06T02:16:00.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:00 smithi017 ceph-mon[29567]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-06T02:16:02.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:02 smithi104 ceph-mon[25739]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:02.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:02 smithi017 ceph-mon[29567]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:02.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:02 smithi017 ceph-mon[24844]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:04.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:04 smithi104 ceph-mon[25739]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:04.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:04.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:04.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:04.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:04 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[24844]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[29567]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:04.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:04 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:05.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:05 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:05.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:05 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:05.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:05.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:05 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:05.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:05 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:05.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:05 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:06.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:06 smithi104 ceph-mon[25739]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:06.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:06 smithi017 ceph-mon[29567]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:06.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:06 smithi017 ceph-mon[24844]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:07.386 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 7 on host 'smithi104' 2024-04-06T02:16:07.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:07 smithi104 ceph-mon[25739]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:07 smithi104 ceph-mon[25739]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:07 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:07.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[29567]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[29567]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:07.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[24844]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[24844]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:16:07.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:07 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:08.035 DEBUG:teuthology.orchestra.run.smithi104:osd.7> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-06T02:16:08.037 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-06T02:16:08.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: osdmap e52: 8 total, 7 up, 8 in 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:08.915 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[37838]: 2024-04-06T02:16:08.647+0000 7fd2edf0c700 -1 osd.7 0 waiting for initial osdmap 2024-04-06T02:16:08.915 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:16:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[37838]: 2024-04-06T02:16:08.654+0000 7fd2e90a4700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:16:08.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: osdmap e52: 8 total, 7 up, 8 in 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: osdmap e52: 8 total, 7 up, 8 in 2024-04-06T02:16:08.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:08.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: from='osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:16:08.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:08 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: osdmap e53: 8 total, 7 up, 8 in 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656] boot 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: osdmap e54: 8 total, 8 up, 8 in 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: osdmap e53: 8 total, 7 up, 8 in 2024-04-06T02:16:09.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656] boot 2024-04-06T02:16:09.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: osdmap e54: 8 total, 8 up, 8 in 2024-04-06T02:16:09.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:09 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-06T02:16:09.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: osdmap e53: 8 total, 7 up, 8 in 2024-04-06T02:16:09.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:09.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: osd.7 [v2:172.21.15.104:6824/3632195656,v1:172.21.15.104:6825/3632195656] boot 2024-04-06T02:16:09.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: osdmap e54: 8 total, 8 up, 8 in 2024-04-06T02:16:09.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:09 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:16:10.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[29567]: purged_snaps scrub starts 2024-04-06T02:16:10.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[29567]: purged_snaps scrub ok 2024-04-06T02:16:10.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[29567]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:10.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1250880201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T02:16:10.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[29567]: osdmap e55: 8 total, 8 up, 8 in 2024-04-06T02:16:10.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[24844]: purged_snaps scrub starts 2024-04-06T02:16:10.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[24844]: purged_snaps scrub ok 2024-04-06T02:16:10.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[24844]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:10.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1250880201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T02:16:10.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:10 smithi017 ceph-mon[24844]: osdmap e55: 8 total, 8 up, 8 in 2024-04-06T02:16:10.838 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1712369769,"num_in_osds":8,"osd_in_since":1712369752,"num_remapped_pgs":1} 2024-04-06T02:16:10.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:16:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:10 smithi104 ceph-mon[25739]: purged_snaps scrub starts 2024-04-06T02:16:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:10 smithi104 ceph-mon[25739]: purged_snaps scrub ok 2024-04-06T02:16:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:10 smithi104 ceph-mon[25739]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-06T02:16:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:10 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1250880201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T02:16:10.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:10 smithi104 ceph-mon[25739]: osdmap e55: 8 total, 8 up, 8 in 2024-04-06T02:16:11.060 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: osdmap e56: 8 total, 8 up, 8 in 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: Detected new or changed devices on smithi104 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi104 to 4272M 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: osdmap e56: 8 total, 8 up, 8 in 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: Detected new or changed devices on smithi104 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.742 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi104 to 4272M 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:12.743 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:12 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.839 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:16:12.840 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":57,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:11:20.015851+0000","modified":"2024-04-06T02:16:12.694138+0000","last_up_change":"2024-04-06T02:16:09.522214+0000","last_in_change":"2024-04-06T02:15:52.831414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:14:13.805254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"72fc82a0-60b7-44f9-a000-ef711644aacb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6803","nonce":669246563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6805","nonce":669246563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6809","nonce":669246563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6807","nonce":669246563}]},"public_addr":"172.21.15.17:6803/669246563","cluster_addr":"172.21.15.17:6805/669246563","heartbeat_back_addr":"172.21.15.17:6809/669246563","heartbeat_front_addr":"172.21.15.17:6807/669246563","state":["exists","up"]},{"osd":1,"uuid":"4ca29e90-938b-4c53-896d-20661c1cd1d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6811","nonce":320210571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6813","nonce":320210571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6817","nonce":320210571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6815","nonce":320210571}]},"public_addr":"172.21.15.17:6811/320210571","cluster_addr":"172.21.15.17:6813/320210571","heartbeat_back_addr":"172.21.15.17:6817/320210571","heartbeat_front_addr":"172.21.15.17:6815/320210571","state":["exists","up"]},{"osd":2,"uuid":"3bb8c263-c05c-488e-9574-310742483a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6819","nonce":1751363685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6821","nonce":1751363685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6825","nonce":1751363685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6823","nonce":1751363685}]},"public_addr":"172.21.15.17:6819/1751363685","cluster_addr":"172.21.15.17:6821/1751363685","heartbeat_back_addr":"172.21.15.17:6825/1751363685","heartbeat_front_addr":"172.21.15.17:6823/1751363685","state":["exists","up"]},{"osd":3,"uuid":"92befe7c-2f6f-4c88-a41b-58cc7dd04c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6827","nonce":2170379733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6829","nonce":2170379733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6833","nonce":2170379733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6831","nonce":2170379733}]},"public_addr":"172.21.15.17:6827/2170379733","cluster_addr":"172.21.15.17:6829/2170379733","heartbeat_back_addr":"172.21.15.17:6833/2170379733","heartbeat_front_addr":"172.21.15.17:6831/2170379733","state":["exists","up"]},{"osd":4,"uuid":"f8b86c23-048b-4913-bc79-b75e88d8425b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6801","nonce":260389348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6803","nonce":260389348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6807","nonce":260389348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6805","nonce":260389348}]},"public_addr":"172.21.15.104:6801/260389348","cluster_addr":"172.21.15.104:6803/260389348","heartbeat_back_addr":"172.21.15.104:6807/260389348","heartbeat_front_addr":"172.21.15.104:6805/260389348","state":["exists","up"]},{"osd":5,"uuid":"ff15491f-631c-40d1-afb0-0612c6bd6c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6809","nonce":3760683088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6811","nonce":3760683088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6815","nonce":3760683088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6813","nonce":3760683088}]},"public_addr":"172.21.15.104:6809/3760683088","cluster_addr":"172.21.15.104:6811/3760683088","heartbeat_back_addr":"172.21.15.104:6815/3760683088","heartbeat_front_addr":"172.21.15.104:6813/3760683088","state":["exists","up"]},{"osd":6,"uuid":"ec70d0cd-b89d-4885-b7b6-e318c10a3bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6817","nonce":579793818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6819","nonce":579793818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6823","nonce":579793818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6821","nonce":579793818}]},"public_addr":"172.21.15.104:6817/579793818","cluster_addr":"172.21.15.104:6819/579793818","heartbeat_back_addr":"172.21.15.104:6823/579793818","heartbeat_front_addr":"172.21.15.104:6821/579793818","state":["exists","up"]},{"osd":7,"uuid":"d4e3423d-e4c3-448d-9e21-c12c85405720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6825","nonce":3632195656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6827","nonce":3632195656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6831","nonce":3632195656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6829","nonce":3632195656}]},"public_addr":"172.21.15.104:6825/3632195656","cluster_addr":"172.21.15.104:6827/3632195656","heartbeat_back_addr":"172.21.15.104:6831/3632195656","heartbeat_front_addr":"172.21.15.104:6829/3632195656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:23.328957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:47.298795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:10.856995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:34.867983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:56.519740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:20.616153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:44.553351+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:16:08.389049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2756296616":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/2917214294":"2024-04-07T02:12:04.649134+0000","172.21.15.17:6800/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/4025515538":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/2438202178":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6800/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:0/1295449500":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/3736517106":"2024-04-07T02:11:42.497272+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: osdmap e56: 8 total, 8 up, 8 in 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: Detected new or changed devices on smithi104 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:16:12.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi104 to 4272M 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:12.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:12 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:13.480 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T02:14:13.805254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-06T02:16:13.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-06T02:16:13.702 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:13.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:13 smithi017 ceph-mon[29567]: osdmap e57: 8 total, 8 up, 8 in 2024-04-06T02:16:13.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:13 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/234515289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:13.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:13 smithi017 ceph-mon[24844]: osdmap e57: 8 total, 8 up, 8 in 2024-04-06T02:16:13.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:13 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/234515289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:14.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:13 smithi104 ceph-mon[25739]: osdmap e57: 8 total, 8 up, 8 in 2024-04-06T02:16:14.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:13 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/234515289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:14.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:14 smithi017 ceph-mon[24844]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:14.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:14 smithi017 ceph-mon[24844]: osdmap e58: 8 total, 8 up, 8 in 2024-04-06T02:16:14.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:14 smithi017 ceph-mon[29567]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:14.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:14 smithi017 ceph-mon[29567]: osdmap e58: 8 total, 8 up, 8 in 2024-04-06T02:16:15.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:14 smithi104 ceph-mon[25739]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:15.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:14 smithi104 ceph-mon[25739]: osdmap e58: 8 total, 8 up, 8 in 2024-04-06T02:16:15.519 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2024-04-06T02:16:15.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:15 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1923789153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:16:15.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:15 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1923789153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:16:16.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:15 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1923789153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:16:16.107 INFO:tasks.cephadm:Adding prometheus.a on smithi104 2024-04-06T02:16:16.107 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi104=a' 2024-04-06T02:16:16.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:16 smithi017 ceph-mon[29567]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:16:16.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:16 smithi017 ceph-mon[24844]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:16:17.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:16 smithi104 ceph-mon[25739]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-06T02:16:18.015 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled prometheus update... 2024-04-06T02:16:18.637 DEBUG:teuthology.orchestra.run.smithi104:prometheus.a> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service 2024-04-06T02:16:18.639 INFO:tasks.cephadm:Adding node-exporter.a on smithi017 2024-04-06T02:16:18.639 INFO:tasks.cephadm:Adding node-exporter.b on smithi104 2024-04-06T02:16:18.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi017=a;smithi104=b' 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:18 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T02:16:18.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:18.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' 2024-04-06T02:16:18.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:18 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T02:16:19.333 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ignoring --setuser ceph since I am not root 2024-04-06T02:16:19.333 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ignoring --setgroup ceph since I am not root 2024-04-06T02:16:19.333 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-mgr[26922]: -- 172.21.15.104:0/3163106818 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605e6f2c340 con 0x5605e6f05000 2024-04-06T02:16:19.490 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setuser ceph since I am not root 2024-04-06T02:16:19.491 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setgroup ceph since I am not root 2024-04-06T02:16:19.491 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mgr[25081]: -- 172.21.15.17:0/3027466520 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55acf2fd6340 con 0x55acf2faf000 2024-04-06T02:16:19.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:19.585+0000 7f458d11c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:16:19.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:19.676+0000 7f458d11c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:16:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-mon[25739]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-mon[25739]: Saving service prometheus spec with placement smithi104=a;count:1 2024-04-06T02:16:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-mon[25739]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T02:16:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:19 smithi104 ceph-mon[25739]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-06T02:16:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[24844]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[24844]: Saving service prometheus spec with placement smithi104=a;count:1 2024-04-06T02:16:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[24844]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T02:16:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[24844]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:19.604+0000 7fbc523c8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:19.695+0000 7fbc523c8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[29567]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[29567]: Saving service prometheus spec with placement smithi104=a;count:1 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[29567]: from='mgr.14150 172.21.15.17:0/4046386290' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T02:16:19.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:19 smithi017 ceph-mon[29567]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-06T02:16:20.601 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.282+0000 7f458d11c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:16:20.601 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.374+0000 7f458d11c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:16:20.601 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.460+0000 7f458d11c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:16:20.601 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.614+0000 7f458d11c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:16:20.620 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.306+0000 7fbc523c8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:16:20.620 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.397+0000 7fbc523c8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:16:20.620 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.482+0000 7fbc523c8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:16:20.620 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.634+0000 7fbc523c8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:16:20.873 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.695+0000 7f458d11c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:16:20.873 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.804+0000 7f458d11c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:16:20.873 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:20.886+0000 7f458d11c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:16:20.904 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.717+0000 7fbc523c8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:16:20.904 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.832+0000 7fbc523c8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:16:21.157 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:20.918+0000 7fbc523c8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:16:21.157 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:21 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:21.083+0000 7fbc523c8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:16:21.157 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:21 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:21.171+0000 7fbc523c8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:16:21.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:21.045+0000 7f458d11c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:16:21.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:21.130+0000 7f458d11c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:16:22.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.134+0000 7f458d11c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:16:22.414 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.262+0000 7f458d11c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:16:22.490 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:22.239+0000 7fbc523c8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:16:22.490 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:22.372+0000 7fbc523c8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:16:22.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.449+0000 7f458d11c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:16:22.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.535+0000 7f458d11c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:16:22.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.668+0000 7f458d11c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:16:22.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:22.569+0000 7fbc523c8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:16:22.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:22.659+0000 7fbc523c8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:16:22.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:22.800+0000 7fbc523c8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:16:23.272 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:22.984+0000 7f458d11c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:16:23.272 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:23.201+0000 7f458d11c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:16:23.429 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:23.126+0000 7fbc523c8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:16:23.430 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:23.355+0000 7fbc523c8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:16:23.430 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:23.443+0000 7fbc523c8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:16:23.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:23.285+0000 7f458d11c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:16:23.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:23.453+0000 7f458d11c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:16:23.740 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:23.618+0000 7fbc523c8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:16:24.634 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:24.357+0000 7f458d11c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:16:24.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:24.572+0000 7fbc523c8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:16:24.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:24.880+0000 7fbc523c8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:16:24.891 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:24.647+0000 7f458d11c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:16:24.891 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:24.741+0000 7f458d11c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:16:24.891 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:24.889+0000 7f458d11c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:16:24.891 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: [06/Apr/2024:02:16:24] ENGINE Bus STARTING 2024-04-06T02:16:25.122 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:24.980+0000 7fbc523c8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:16:25.122 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:25.136+0000 7fbc523c8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:16:25.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: CherryPy Checker: 2024-04-06T02:16:25.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: The Application mounted at '' has an empty config. 2024-04-06T02:16:25.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: 2024-04-06T02:16:25.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: [06/Apr/2024:02:16:25] ENGINE Serving on http://:::9283 2024-04-06T02:16:25.164 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: [06/Apr/2024:02:16:25] ENGINE Bus STARTED 2024-04-06T02:16:25.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:25] ENGINE Bus STARTING 2024-04-06T02:16:25.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: CherryPy Checker: 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: Standby manager daemon x restarted 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: Active manager daemon y restarted 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: Activating manager daemon y 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[29567]: osdmap e59: 8 total, 8 up, 8 in 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: Standby manager daemon x restarted 2024-04-06T02:16:25.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: Standby manager daemon x started 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: Active manager daemon y restarted 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: Activating manager daemon y 2024-04-06T02:16:25.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-mon[24844]: osdmap e59: 8 total, 8 up, 8 in 2024-04-06T02:16:25.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: Standby manager daemon x restarted 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/660037163' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: Active manager daemon y restarted 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: Activating manager daemon y 2024-04-06T02:16:25.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:25 smithi104 ceph-mon[25739]: osdmap e59: 8 total, 8 up, 8 in 2024-04-06T02:16:26.118 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: The Application mounted at '' has an empty config. 2024-04-06T02:16:26.118 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:16:26.119 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:25] ENGINE Serving on http://:::9283 2024-04-06T02:16:26.119 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:25] ENGINE Bus STARTED 2024-04-06T02:16:26.541 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled node-exporter update... 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: mgrmap e17: y(active, starting, since 0.386756s), standbys: x 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:16:26.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: Manager daemon y is now available 2024-04-06T02:16:26.780 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: mgrmap e17: y(active, starting, since 0.386756s), standbys: x 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:16:26.781 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: Manager daemon y is now available 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.782 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:26.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:26.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:26.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:26.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: mgrmap e17: y(active, starting, since 0.386756s), standbys: x 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:16:26.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: Manager daemon y is now available 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:26.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:26.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:26.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:16:26.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:16:27.240 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:26] ENGINE Bus STARTING 2024-04-06T02:16:27.240 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:26] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:16:27.240 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:16:26] ENGINE Bus STARTED 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: mgrmap e18: y(active, since 1.39388s), standbys: x 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: Saving service node-exporter spec with placement smithi017=a;smithi104=b;count:2 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: [06/Apr/2024:02:16:26] ENGINE Bus STARTING 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: [06/Apr/2024:02:16:26] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: [06/Apr/2024:02:16:26] ENGINE Bus STARTED 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.655 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.677 DEBUG:teuthology.orchestra.run.smithi017:node-exporter.a> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service 2024-04-06T02:16:27.679 DEBUG:teuthology.orchestra.run.smithi104:node-exporter.b> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service 2024-04-06T02:16:27.681 INFO:tasks.cephadm:Adding alertmanager.a on smithi017 2024-04-06T02:16:27.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi017=a' 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: mgrmap e18: y(active, since 1.39388s), standbys: x 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: Saving service node-exporter spec with placement smithi017=a;smithi104=b;count:2 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: [06/Apr/2024:02:16:26] ENGINE Bus STARTING 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: [06/Apr/2024:02:16:26] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: [06/Apr/2024:02:16:26] ENGINE Bus STARTED 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: mgrmap e18: y(active, since 1.39388s), standbys: x 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: Saving service node-exporter spec with placement smithi017=a;smithi104=b;count:2 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: [06/Apr/2024:02:16:26] ENGINE Bus STARTING 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: [06/Apr/2024:02:16:26] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: [06/Apr/2024:02:16:26] ENGINE Bus STARTED 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:27.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:28.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:28 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:29.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-06T02:16:29.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:16:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:28 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:29.910 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled alertmanager update... 2024-04-06T02:16:30.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:16:30.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:30 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:30 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:30.644 DEBUG:teuthology.orchestra.run.smithi017:alertmanager.a> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@alertmanager.a.service 2024-04-06T02:16:30.647 INFO:tasks.cephadm:Adding grafana.a on smithi104 2024-04-06T02:16:30.647 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph orch apply grafana '1;smithi104=a' 2024-04-06T02:16:31.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:31 smithi104 ceph-mon[25739]: Deploying daemon prometheus.a on smithi104 2024-04-06T02:16:31.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:31 smithi104 ceph-mon[25739]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:31.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:31 smithi104 ceph-mon[25739]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:31.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:31 smithi104 ceph-mon[25739]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-04-06T02:16:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[29567]: Deploying daemon prometheus.a on smithi104 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[29567]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[29567]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[29567]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[24844]: Deploying daemon prometheus.a on smithi104 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[24844]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[24844]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:31 smithi017 ceph-mon[24844]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-04-06T02:16:33.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:33 smithi104 ceph-mon[25739]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:33.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:33 smithi017 ceph-mon[29567]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:33.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:33 smithi017 ceph-mon[24844]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:33.628 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled grafana update... 2024-04-06T02:16:34.766 DEBUG:teuthology.orchestra.run.smithi104:grafana.a> sudo journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service 2024-04-06T02:16:34.768 INFO:tasks.cephadm:Setting up client nodes... 2024-04-06T02:16:34.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T02:16:34.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:34 smithi104 ceph-mon[25739]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:34.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:34 smithi104 ceph-mon[25739]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:34.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:34 smithi104 ceph-mon[25739]: Saving service grafana spec with placement smithi104=a;count:1 2024-04-06T02:16:34.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:34.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[29567]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:34.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[29567]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:34.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[29567]: Saving service grafana spec with placement smithi104=a;count:1 2024-04-06T02:16:34.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:34.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[24844]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:34.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[24844]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:16:34.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[24844]: Saving service grafana spec with placement smithi104=a;count:1 2024-04-06T02:16:34.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:36.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:36 smithi104 ceph-mon[25739]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:36.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:36 smithi017 ceph-mon[29567]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:36.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:36 smithi017 ceph-mon[24844]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:37.008 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-04-06T02:16:37.009 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCFsBBmjkI1ARAAcL/J3Us2S5mdEOhsMDrRxA== 2024-04-06T02:16:37.770 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-04-06T02:16:37.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-06T02:16:37.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T02:16:37.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T02:16:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:37 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1819315065' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:37 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:37 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:37.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1819315065' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:37.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1819315065' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:37.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:37 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:39 smithi017 ceph-mon[29567]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:39.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:39 smithi017 ceph-mon[24844]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:39.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:39 smithi104 ceph-mon[25739]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:40.878 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:40 smithi104 ceph-mon[25739]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:40.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:40 smithi017 ceph-mon[29567]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:40.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:40 smithi017 ceph-mon[24844]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:41.532 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:41 smithi104 systemd[1]: Starting Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:16:42.331 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-04-06T02:16:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCKsBBm7C9LEhAAAMTZ4IMtnSRyYSGBBSfi2A== 2024-04-06T02:16:42.530 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 podman[40789]: 2024-04-06 02:16:42.082137575 +0000 UTC m=+0.018932431 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-06T02:16:42.530 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 podman[40789]: 2024-04-06 02:16:42.199780052 +0000 UTC m=+0.136574947 container create 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:42.530 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 podman[40789]: 2024-04-06 02:16:42.544113977 +0000 UTC m=+0.480908823 container init 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:42.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-mon[25739]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:42.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.104:0/2812342119' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:42.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 podman[40789]: 2024-04-06 02:16:42.547043265 +0000 UTC m=+0.483838112 container start 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi104 (none))" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.571Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.572Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.573Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.574Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.575Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.575Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.661µs 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.575Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.576Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.576Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.074µs wal_replay_duration=497.498µs total_replay_duration=538.22µs 2024-04-06T02:16:42.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.578Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.578Z caller=main.go:947 level=info msg="TSDB started" 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.578Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.592Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.30253ms db_storage=1.253µs remote_storage=1.1µs web_handler=253ns query_engine=604ns scrape=609.967µs scrape_sd=17.039µs notify=886ns notify_sd=1.394µs rules=13.495786ms 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:16:42.592Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 bash[40789]: 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c 2024-04-06T02:16:42.916 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:16:42 smithi104 systemd[1]: Started Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:16:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[24844]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.104:0/2812342119' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[29567]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.104:0/2812342119' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:16:42.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:16:43.272 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-06T02:16:43.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-06T02:16:43.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-06T02:16:43.316 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T02:16:43.316 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-06T02:16:43.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-06T02:16:43.580 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:43.676 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:43 smithi017 bash[44231]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-06T02:16:43.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:43 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:43.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:43 smithi017 ceph-mon[24844]: Deploying daemon node-exporter.a on smithi017 2024-04-06T02:16:43.929 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:43 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:43.929 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:43 smithi017 ceph-mon[29567]: Deploying daemon node-exporter.a on smithi017 2024-04-06T02:16:43.929 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:43 smithi017 bash[44231]: Getting image source signatures 2024-04-06T02:16:43.929 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:43 smithi017 bash[44231]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-06T02:16:43.929 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:43 smithi017 bash[44231]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-06T02:16:43.929 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:43 smithi017 bash[44231]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-06T02:16:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:43 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:44.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:43 smithi104 ceph-mon[25739]: Deploying daemon node-exporter.a on smithi017 2024-04-06T02:16:44.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:44 smithi017 ceph-mon[29567]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:44.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:44 smithi017 ceph-mon[24844]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:44 smithi104 ceph-mon[25739]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:45.740 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:45 smithi017 bash[44231]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-06T02:16:46.090 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:45 smithi017 bash[44231]: Writing manifest to image destination 2024-04-06T02:16:46.436 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:16:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[24844]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1838111134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T02:16:46.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[29567]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:46.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:46.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1838111134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T02:16:46.741 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 podman[44231]: 2024-04-06 02:16:46.333481629 +0000 UTC m=+2.859508515 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-06T02:16:46.741 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 podman[44231]: 2024-04-06 02:16:46.457168498 +0000 UTC m=+2.983195373 container create db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:46.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:46 smithi104 ceph-mon[25739]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:46.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:46 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:46.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:46 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1838111134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 podman[44231]: 2024-04-06 02:16:46.902913582 +0000 UTC m=+3.428940467 container init db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 podman[44231]: 2024-04-06 02:16:46.905797397 +0000 UTC m=+3.431824275 container start db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.909Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.909Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.910Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.910Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.910Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=arp 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-06T02:16:47.038 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=edac 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-06T02:16:47.039 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=os 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-06T02:16:47.040 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=stat 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=time 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=uname 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[44411]: ts=2024-04-06T02:16:46.911Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-06T02:16:47.041 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 bash[44231]: db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 2024-04-06T02:16:47.042 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:16:46 smithi017 systemd[1]: Started Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:16:47.573 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":19,"active_gid":14430,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":611295603},{"type":"v1","addr":"172.21.15.17:6801","nonce":611295603}]},"active_addr":"172.21.15.17:6801/611295603","active_change":"2024-04-06T02:16:25.150453+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34334,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.17:8443/","prometheus":"http://172.21.15.17:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2724707203}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1233898327}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3771182315}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":136955192}]}]}} 2024-04-06T02:16:47.578 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-06T02:16:47.583 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T02:16:47.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:16:47.802 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:48.052 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:47 smithi104 bash[41078]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-06T02:16:48.179 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:48 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:48.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:48 smithi017 ceph-mon[29567]: Deploying daemon node-exporter.b on smithi104 2024-04-06T02:16:48.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:48 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:48.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:48 smithi017 ceph-mon[24844]: Deploying daemon node-exporter.b on smithi104 2024-04-06T02:16:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:48 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:48 smithi104 ceph-mon[25739]: Deploying daemon node-exporter.b on smithi104 2024-04-06T02:16:48.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:48 smithi104 bash[41078]: Getting image source signatures 2024-04-06T02:16:48.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:48 smithi104 bash[41078]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-06T02:16:48.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:48 smithi104 bash[41078]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-06T02:16:48.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:48 smithi104 bash[41078]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-06T02:16:49.283 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:49 smithi017 ceph-mon[24844]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:49.283 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:49 smithi017 ceph-mon[29567]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:49 smithi104 ceph-mon[25739]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:49.772 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:16:49.772 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":59,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:11:20.015851+0000","modified":"2024-04-06T02:16:25.150032+0000","last_up_change":"2024-04-06T02:16:09.522214+0000","last_in_change":"2024-04-06T02:15:52.831414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:14:13.805254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"72fc82a0-60b7-44f9-a000-ef711644aacb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6803","nonce":669246563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6805","nonce":669246563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6809","nonce":669246563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6807","nonce":669246563}]},"public_addr":"172.21.15.17:6803/669246563","cluster_addr":"172.21.15.17:6805/669246563","heartbeat_back_addr":"172.21.15.17:6809/669246563","heartbeat_front_addr":"172.21.15.17:6807/669246563","state":["exists","up"]},{"osd":1,"uuid":"4ca29e90-938b-4c53-896d-20661c1cd1d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6811","nonce":320210571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6813","nonce":320210571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6817","nonce":320210571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6815","nonce":320210571}]},"public_addr":"172.21.15.17:6811/320210571","cluster_addr":"172.21.15.17:6813/320210571","heartbeat_back_addr":"172.21.15.17:6817/320210571","heartbeat_front_addr":"172.21.15.17:6815/320210571","state":["exists","up"]},{"osd":2,"uuid":"3bb8c263-c05c-488e-9574-310742483a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6819","nonce":1751363685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6821","nonce":1751363685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6825","nonce":1751363685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6823","nonce":1751363685}]},"public_addr":"172.21.15.17:6819/1751363685","cluster_addr":"172.21.15.17:6821/1751363685","heartbeat_back_addr":"172.21.15.17:6825/1751363685","heartbeat_front_addr":"172.21.15.17:6823/1751363685","state":["exists","up"]},{"osd":3,"uuid":"92befe7c-2f6f-4c88-a41b-58cc7dd04c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6827","nonce":2170379733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6829","nonce":2170379733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6833","nonce":2170379733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6831","nonce":2170379733}]},"public_addr":"172.21.15.17:6827/2170379733","cluster_addr":"172.21.15.17:6829/2170379733","heartbeat_back_addr":"172.21.15.17:6833/2170379733","heartbeat_front_addr":"172.21.15.17:6831/2170379733","state":["exists","up"]},{"osd":4,"uuid":"f8b86c23-048b-4913-bc79-b75e88d8425b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6801","nonce":260389348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6803","nonce":260389348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6807","nonce":260389348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6805","nonce":260389348}]},"public_addr":"172.21.15.104:6801/260389348","cluster_addr":"172.21.15.104:6803/260389348","heartbeat_back_addr":"172.21.15.104:6807/260389348","heartbeat_front_addr":"172.21.15.104:6805/260389348","state":["exists","up"]},{"osd":5,"uuid":"ff15491f-631c-40d1-afb0-0612c6bd6c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6809","nonce":3760683088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6811","nonce":3760683088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6815","nonce":3760683088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6813","nonce":3760683088}]},"public_addr":"172.21.15.104:6809/3760683088","cluster_addr":"172.21.15.104:6811/3760683088","heartbeat_back_addr":"172.21.15.104:6815/3760683088","heartbeat_front_addr":"172.21.15.104:6813/3760683088","state":["exists","up"]},{"osd":6,"uuid":"ec70d0cd-b89d-4885-b7b6-e318c10a3bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6817","nonce":579793818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6819","nonce":579793818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6823","nonce":579793818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6821","nonce":579793818}]},"public_addr":"172.21.15.104:6817/579793818","cluster_addr":"172.21.15.104:6819/579793818","heartbeat_back_addr":"172.21.15.104:6823/579793818","heartbeat_front_addr":"172.21.15.104:6821/579793818","state":["exists","up"]},{"osd":7,"uuid":"d4e3423d-e4c3-448d-9e21-c12c85405720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6825","nonce":3632195656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6827","nonce":3632195656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6831","nonce":3632195656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6829","nonce":3632195656}]},"public_addr":"172.21.15.104:6825/3632195656","cluster_addr":"172.21.15.104:6827/3632195656","heartbeat_back_addr":"172.21.15.104:6831/3632195656","heartbeat_front_addr":"172.21.15.104:6829/3632195656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:23.328957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:47.298795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:10.856995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:34.867983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:56.519740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:20.616153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:44.553351+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:16:08.389049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2438695043":"2024-04-07T02:16:25.149983+0000","172.21.15.17:6801/397925205":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/551721499":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/2756296616":"2024-04-07T02:12:04.649134+0000","172.21.15.17:6800/397925205":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/2917214294":"2024-04-07T02:12:04.649134+0000","172.21.15.17:6800/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/4025515538":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/2438202178":"2024-04-07T02:11:42.497272+0000","172.21.15.17:0/1418949283":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/3265016438":"2024-04-07T02:16:25.149983+0000","172.21.15.17:6800/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:0/1295449500":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/3736517106":"2024-04-07T02:11:42.497272+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:16:50.135 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:49 smithi104 bash[41078]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-06T02:16:50.135 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:49 smithi104 bash[41078]: Writing manifest to image destination 2024-04-06T02:16:50.414 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:16:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:16:50.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 podman[41078]: 2024-04-06 02:16:50.133539693 +0000 UTC m=+2.264003604 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-06T02:16:50.414 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 podman[41078]: 2024-04-06 02:16:50.248797 +0000 UTC m=+2.379260953 container create ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:16:50.515 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T02:16:50.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:16:50.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-mon[25739]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:50.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2647198627' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:50.671 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 podman[41078]: 2024-04-06 02:16:50.434293141 +0000 UTC m=+2.564757051 container init ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:16:50.671 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 podman[41078]: 2024-04-06 02:16:50.437272804 +0000 UTC m=+2.567736714 container start ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.440Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.440Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.440Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.440Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=arp 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-06T02:16:50.672 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=edac 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-06T02:16:50.673 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=os 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=stat 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-06T02:16:50.674 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=time 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.441Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=node_exporter.go:115 level=info collector=uname 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[41129]: ts=2024-04-06T02:16:50.442Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 bash[41078]: ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 2024-04-06T02:16:50.675 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:16:50 smithi104 systemd[1]: Started Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:16:50.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:50 smithi017 ceph-mon[29567]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:50.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:50 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2647198627' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:50.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:50 smithi017 ceph-mon[24844]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:50.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:50 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2647198627' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:50.746 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:51.863 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:51.864 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:51.864 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:51 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:51.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:51 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:51.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:51 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:16:51.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:51 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:16:51.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:51 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:16:52.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:52 smithi017 ceph-mon[29567]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:52.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:52 smithi017 ceph-mon[24844]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:52.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:52 smithi104 ceph-mon[25739]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:53.180 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:16:53.180 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":59,"fsid":"dbfad14a-f3ba-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:11:20.015851+0000","modified":"2024-04-06T02:16:25.150032+0000","last_up_change":"2024-04-06T02:16:09.522214+0000","last_in_change":"2024-04-06T02:15:52.831414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:14:13.805254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"72fc82a0-60b7-44f9-a000-ef711644aacb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6803","nonce":669246563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6805","nonce":669246563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6809","nonce":669246563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":669246563},{"type":"v1","addr":"172.21.15.17:6807","nonce":669246563}]},"public_addr":"172.21.15.17:6803/669246563","cluster_addr":"172.21.15.17:6805/669246563","heartbeat_back_addr":"172.21.15.17:6809/669246563","heartbeat_front_addr":"172.21.15.17:6807/669246563","state":["exists","up"]},{"osd":1,"uuid":"4ca29e90-938b-4c53-896d-20661c1cd1d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6811","nonce":320210571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6813","nonce":320210571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6817","nonce":320210571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":320210571},{"type":"v1","addr":"172.21.15.17:6815","nonce":320210571}]},"public_addr":"172.21.15.17:6811/320210571","cluster_addr":"172.21.15.17:6813/320210571","heartbeat_back_addr":"172.21.15.17:6817/320210571","heartbeat_front_addr":"172.21.15.17:6815/320210571","state":["exists","up"]},{"osd":2,"uuid":"3bb8c263-c05c-488e-9574-310742483a34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6819","nonce":1751363685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6821","nonce":1751363685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6825","nonce":1751363685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1751363685},{"type":"v1","addr":"172.21.15.17:6823","nonce":1751363685}]},"public_addr":"172.21.15.17:6819/1751363685","cluster_addr":"172.21.15.17:6821/1751363685","heartbeat_back_addr":"172.21.15.17:6825/1751363685","heartbeat_front_addr":"172.21.15.17:6823/1751363685","state":["exists","up"]},{"osd":3,"uuid":"92befe7c-2f6f-4c88-a41b-58cc7dd04c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6827","nonce":2170379733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6829","nonce":2170379733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6833","nonce":2170379733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2170379733},{"type":"v1","addr":"172.21.15.17:6831","nonce":2170379733}]},"public_addr":"172.21.15.17:6827/2170379733","cluster_addr":"172.21.15.17:6829/2170379733","heartbeat_back_addr":"172.21.15.17:6833/2170379733","heartbeat_front_addr":"172.21.15.17:6831/2170379733","state":["exists","up"]},{"osd":4,"uuid":"f8b86c23-048b-4913-bc79-b75e88d8425b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6801","nonce":260389348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6803","nonce":260389348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6807","nonce":260389348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":260389348},{"type":"v1","addr":"172.21.15.104:6805","nonce":260389348}]},"public_addr":"172.21.15.104:6801/260389348","cluster_addr":"172.21.15.104:6803/260389348","heartbeat_back_addr":"172.21.15.104:6807/260389348","heartbeat_front_addr":"172.21.15.104:6805/260389348","state":["exists","up"]},{"osd":5,"uuid":"ff15491f-631c-40d1-afb0-0612c6bd6c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6809","nonce":3760683088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6811","nonce":3760683088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6815","nonce":3760683088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3760683088},{"type":"v1","addr":"172.21.15.104:6813","nonce":3760683088}]},"public_addr":"172.21.15.104:6809/3760683088","cluster_addr":"172.21.15.104:6811/3760683088","heartbeat_back_addr":"172.21.15.104:6815/3760683088","heartbeat_front_addr":"172.21.15.104:6813/3760683088","state":["exists","up"]},{"osd":6,"uuid":"ec70d0cd-b89d-4885-b7b6-e318c10a3bd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6817","nonce":579793818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6819","nonce":579793818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6823","nonce":579793818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":579793818},{"type":"v1","addr":"172.21.15.104:6821","nonce":579793818}]},"public_addr":"172.21.15.104:6817/579793818","cluster_addr":"172.21.15.104:6819/579793818","heartbeat_back_addr":"172.21.15.104:6823/579793818","heartbeat_front_addr":"172.21.15.104:6821/579793818","state":["exists","up"]},{"osd":7,"uuid":"d4e3423d-e4c3-448d-9e21-c12c85405720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6825","nonce":3632195656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6827","nonce":3632195656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6831","nonce":3632195656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3632195656},{"type":"v1","addr":"172.21.15.104:6829","nonce":3632195656}]},"public_addr":"172.21.15.104:6825/3632195656","cluster_addr":"172.21.15.104:6827/3632195656","heartbeat_back_addr":"172.21.15.104:6831/3632195656","heartbeat_front_addr":"172.21.15.104:6829/3632195656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:23.328957+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:13:47.298795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:10.856995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:34.867983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:14:56.519740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:20.616153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:15:44.553351+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:16:08.389049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2438695043":"2024-04-07T02:16:25.149983+0000","172.21.15.17:6801/397925205":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/551721499":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/2756296616":"2024-04-07T02:12:04.649134+0000","172.21.15.17:6800/397925205":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/2917214294":"2024-04-07T02:12:04.649134+0000","172.21.15.17:6800/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/4025515538":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/2438202178":"2024-04-07T02:11:42.497272+0000","172.21.15.17:0/1418949283":"2024-04-07T02:16:25.149983+0000","172.21.15.17:0/3265016438":"2024-04-07T02:16:25.149983+0000","172.21.15.17:6800/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/2138725502":"2024-04-07T02:11:42.497272+0000","172.21.15.17:0/1295449500":"2024-04-07T02:11:42.497272+0000","172.21.15.17:6801/4184866883":"2024-04-07T02:12:04.649134+0000","172.21.15.17:0/3736517106":"2024-04-07T02:11:42.497272+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:16:53.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:53 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3703732264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:53.924 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:53 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3703732264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:53.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:53 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3703732264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:16:53.948 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T02:16:53.948 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T02:16:53.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T02:16:53.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T02:16:53.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T02:16:53.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T02:16:53.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-06T02:16:53.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-06T02:16:54.544 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:54.545 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:54.545 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:54.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:54 smithi017 ceph-mon[29567]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:54.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:54 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:54.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:54 smithi017 ceph-mon[24844]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:54.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:54 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:54.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:54 smithi104 ceph-mon[25739]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:54.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:54 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.143 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:56.143 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:56.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:55 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:55 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:55 smithi104 ceph-mon[25739]: Deploying daemon alertmanager.a on smithi017 2024-04-06T02:16:56.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:55 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[29567]: Deploying daemon alertmanager.a on smithi017 2024-04-06T02:16:56.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[24844]: Deploying daemon alertmanager.a on smithi017 2024-04-06T02:16:56.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:55 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:16:56.871 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:16:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:16:56] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-04-06T02:16:57.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:56 smithi104 ceph-mon[25739]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:57.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:56 smithi017 ceph-mon[29567]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:57.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:56 smithi017 ceph-mon[24844]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:57.776 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:59.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:16:58 smithi104 ceph-mon[25739]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:59.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:16:58 smithi017 ceph-mon[29567]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:59.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:16:58 smithi017 ceph-mon[24844]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:16:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:16:59.633 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:00.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:00.720 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:00 smithi017 ceph-mon[29567]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:00.720 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:00 smithi017 ceph-mon[24844]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:00.748 INFO:teuthology.orchestra.run.smithi017.stdout:231928233995 2024-04-06T02:17:00.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-06T02:17:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:00 smithi104 ceph-mon[25739]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:02.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:02 smithi104 ceph-mon[25739]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:02.982 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:02 smithi017 ceph-mon[29567]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:02.983 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:02 smithi017 ceph-mon[24844]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:04.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:04 smithi104 ceph-mon[25739]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:04.922 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:04 smithi017 ceph-mon[24844]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:04.923 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:04 smithi017 ceph-mon[29567]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:06.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:06 smithi017 ceph-mon[29567]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:06.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:06 smithi017 ceph-mon[24844]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:06.740 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:06] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-04-06T02:17:06.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:06 smithi104 ceph-mon[25739]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:08.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:08 smithi017 ceph-mon[29567]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:08.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:08 smithi017 ceph-mon[24844]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:08.865 INFO:teuthology.orchestra.run.smithi017.stdout:197568495633 2024-04-06T02:17:08.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-06T02:17:08.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:08.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:08 smithi104 ceph-mon[25739]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:10.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:10 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:10.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:10 smithi017 ceph-mon[29567]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:10.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:10 smithi017 ceph-mon[24844]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:10.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:10 smithi104 ceph-mon[25739]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:11.782 INFO:teuthology.orchestra.run.smithi017.stdout:34359738414 2024-04-06T02:17:11.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T02:17:12.839 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:12 smithi017 ceph-mon[29567]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:12.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:12 smithi017 ceph-mon[24844]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:12.864 INFO:teuthology.orchestra.run.smithi017.stdout:128849018908 2024-04-06T02:17:12.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T02:17:12.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:12 smithi104 ceph-mon[25739]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:14.069 INFO:teuthology.orchestra.run.smithi017.stdout:107374182432 2024-04-06T02:17:14.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T02:17:14.544 INFO:teuthology.orchestra.run.smithi017.stdout:55834574889 2024-04-06T02:17:14.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T02:17:14.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:14 smithi017 ceph-mon[29567]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:14.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:14 smithi017 ceph-mon[24844]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:14 smithi104 ceph-mon[25739]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:15.585 INFO:teuthology.orchestra.run.smithi017.stdout:77309411366 2024-04-06T02:17:15.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T02:17:16.190 INFO:teuthology.orchestra.run.smithi017.stdout:163208757273 2024-04-06T02:17:16.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T02:17:16.556 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:16.804 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:16 smithi017 ceph-mon[29567]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:16.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:16 smithi017 ceph-mon[24844]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:16.804 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:16] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-04-06T02:17:16.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:16 smithi104 ceph-mon[25739]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:17.454 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:17.455 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:17.455 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:17.456 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:18.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:18 smithi104 ceph-mon[25739]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:18.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:18 smithi017 ceph-mon[24844]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:18.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:18 smithi017 ceph-mon[29567]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:20.410 INFO:teuthology.orchestra.run.smithi017.stdout:231928233999 2024-04-06T02:17:20.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:20.616 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:20.617 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:20.712 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:20 smithi017 ceph-mon[29567]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:20.712 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:20 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1789758961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T02:17:20.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:20 smithi017 ceph-mon[24844]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:20.713 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:20 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1789758961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T02:17:20.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:20 smithi104 ceph-mon[25739]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:20.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:20 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1789758961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T02:17:22.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:22 smithi017 ceph-mon[29567]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:22.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:22 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4200774317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T02:17:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:22 smithi017 ceph-mon[24844]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:22 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4200774317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T02:17:22.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:22 smithi104 ceph-mon[25739]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:22.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:22 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4200774317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T02:17:24.101 INFO:teuthology.orchestra.run.smithi017.stdout:197568495636 2024-04-06T02:17:24.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233995 got 231928233999 for osd.7 2024-04-06T02:17:24.569 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:24.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:24 smithi017 ceph-mon[24844]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:24.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:24 smithi017 ceph-mon[29567]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:24.836 INFO:teuthology.orchestra.run.smithi017.stdout:128849018911 2024-04-06T02:17:24.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:24 smithi104 ceph-mon[25739]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:25.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:25 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3484021622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T02:17:25.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:25 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3484021622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T02:17:25.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:25 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3484021622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T02:17:26.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[24844]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:26.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:17:26.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:17:26.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:26 smithi104 ceph-mon[25739]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:26.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:17:26.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:17:26.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[29567]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:26.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:17:26.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:17:26.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:26] "GET /metrics HTTP/1.1" 200 191483 "" "Prometheus/2.33.4" 2024-04-06T02:17:27.258 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495633 got 197568495636 for osd.6 2024-04-06T02:17:27.258 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:27.742 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018911 for osd.4 2024-04-06T02:17:27.742 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:28.741 INFO:teuthology.orchestra.run.smithi017.stdout:55834574893 2024-04-06T02:17:28.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:28 smithi104 ceph-mon[25739]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:28.931 INFO:teuthology.orchestra.run.smithi017.stdout:34359738418 2024-04-06T02:17:28.931 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:28 smithi017 ceph-mon[29567]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:28.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:28 smithi017 ceph-mon[24844]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:29.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:29 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2071044096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T02:17:29.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:29 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2329193043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T02:17:29.950 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:29 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2071044096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T02:17:29.951 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:29 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2329193043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T02:17:29.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:29 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2071044096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T02:17:29.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:29 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2329193043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T02:17:30.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:30.735 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:30 smithi017 ceph-mon[29567]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:30.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:30 smithi017 ceph-mon[24844]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:30.759 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574893 for osd.1 2024-04-06T02:17:30.759 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:30.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:30 smithi104 ceph-mon[25739]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:31.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738418 for osd.0 2024-04-06T02:17:31.160 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:32.868 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:32 smithi017 ceph-mon[29567]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:32.868 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:32 smithi017 ceph-mon[24844]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:32.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:32 smithi104 ceph-mon[25739]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout:163208757277 2024-04-06T02:17:33.291 INFO:teuthology.orchestra.run.smithi017.stdout:77309411371 2024-04-06T02:17:33.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:33 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2644112525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T02:17:33.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:33 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3494303809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T02:17:33.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:33 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2644112525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T02:17:33.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:33 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3494303809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T02:17:33.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:33 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2644112525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T02:17:33.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:33 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3494303809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T02:17:34.092 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411371 for osd.2 2024-04-06T02:17:34.092 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:34.529 INFO:teuthology.orchestra.run.smithi017.stdout:107374182437 2024-04-06T02:17:34.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757277 for osd.5 2024-04-06T02:17:34.844 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:34.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:34 smithi104 ceph-mon[25739]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:34.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:34 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4231157295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T02:17:34.914 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:34 smithi017 ceph-mon[24844]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:34.914 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:34 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4231157295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T02:17:34.914 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:34 smithi017 ceph-mon[29567]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:34.914 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:34 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4231157295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T02:17:36.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182437 for osd.3 2024-04-06T02:17:36.069 DEBUG:teuthology.parallel:result is None 2024-04-06T02:17:36.069 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T02:17:36.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T02:17:36.288 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: Deploying daemon grafana.a on smithi104 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: Deploying daemon grafana.a on smithi104 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:36.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: Deploying daemon grafana.a on smithi104 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:36.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:36 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:36.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:36] "GET /metrics HTTP/1.1" 200 191483 "" "Prometheus/2.33.4" 2024-04-06T02:17:37.428 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:37 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[47617]: level=info ts=2024-04-06T02:17:37.340Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00025362s 2024-04-06T02:17:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:17:38.085 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-04-06T02:17:38.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:38 smithi017 ceph-mon[29567]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:38.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:38 smithi017 ceph-mon[29567]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:38 smithi017 ceph-mon[24844]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:38 smithi017 ceph-mon[24844]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:38.838 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-06T02:17:37.554334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48888,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912328,"statfs":{"total":767960285184,"available":767910223872,"internally_reserved":0,"allocated":6217728,"data_stored":3487160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002179"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-06T02:16:25.633486+0000","last_change":"2024-04-06T02:16:13.704449+0000","last_active":"2024-04-06T02:16:25.633486+0000","last_peered":"2024-04-06T02:16:25.633486+0000","last_clean":"2024-04-06T02:16:25.633486+0000","last_became_active":"2024-04-06T02:16:13.704043+0000","last_became_peered":"2024-04-06T02:16:13.704043+0000","last_unstale":"2024-04-06T02:16:25.633486+0000","last_undegraded":"2024-04-06T02:16:25.633486+0000","last_fullsized":"2024-04-06T02:16:25.633486+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T02:14:13.805654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T02:14:13.805654+0000","last_clean_scrub_stamp":"2024-04-06T02:14:13.805654+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:30.706142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.63100000000000001}]},{"osd":3,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.40100000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.505}]},{"osd":6,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52300000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.439},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38800000000000001,"15min":0.36699999999999999},"min":{"1min":0.26800000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Apr 6 02:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.441},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.317},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40200000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79400000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.37},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Apr 6 02:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 02:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 02:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.33100000000000002},"min":{"1min":0.26400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61199999999999999,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Apr 6 02:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39500000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.248,"15min":0.248},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.502}]},{"osd":3,"last update":"Sat Apr 6 02:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.46600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Apr 6 02:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73999999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 6 02:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70399999999999996}]},{"osd":6,"last update":"Sat Apr 6 02:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.622},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Sat Apr 6 02:17:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38500000000000001,"15min":0.38600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 02:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58099999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Apr 6 02:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.29499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.26400000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.38800000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77100000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.41499999999999998}]},{"osd":6,"last update":"Sat Apr 6 02:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44600000000000001}]},{"osd":7,"last update":"Sat Apr 6 02:17:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48399999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.255},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39500000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.314,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57699999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.34799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.46600000000000003},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.315}]},{"osd":4,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77100000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.49299999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.76500000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Apr 6 02:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91100000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":0.73599999999999999,"15min":0.64600000000000002},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":14.262,"5min":14.262,"15min":14.262},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Apr 6 02:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 02:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65800000000000003,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.65300000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56499999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.75,"15min":0.75},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.5}]},{"osd":3,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53900000000000003,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.504,"5min":1.276,"15min":1.4039999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65200000000000002,"5min":18.692,"15min":18.692},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Apr 6 02:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.38,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.44,"15min":0.42599999999999999},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33400000000000002}]},{"osd":6,"last update":"Sat Apr 6 02:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35699999999999998}]},{"osd":7,"last update":"Sat Apr 6 02:17:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.38100000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":236988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.7610000000000001,"5min":5.7610000000000001,"15min":5.7610000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.42299999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69199999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.64600000000000002}]},{"osd":2,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70699999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.73899999999999999}]},{"osd":3,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.753,"15min":0.753},"last":0.752},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56799999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.41499999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64700000000000002,"5min":0.877,"15min":0.877},"last":0.55100000000000005}]},{"osd":6,"last update":"Sat Apr 6 02:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Apr 6 02:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T02:17:38.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T02:17:38.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:38 smithi104 ceph-mon[25739]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:38.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:38 smithi104 ceph-mon[25739]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:39.065 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:40.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:40.794 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:40 smithi017 ceph-mon[29567]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:40.795 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:40 smithi017 ceph-mon[24844]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:40.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:40 smithi104 ceph-mon[25739]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:17:40.968 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-04-06T02:17:41.573 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-06T02:17:39.554639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48888,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912328,"statfs":{"total":767960285184,"available":767910223872,"internally_reserved":0,"allocated":6217728,"data_stored":3487160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002121"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-06T02:16:25.633486+0000","last_change":"2024-04-06T02:16:13.704449+0000","last_active":"2024-04-06T02:16:25.633486+0000","last_peered":"2024-04-06T02:16:25.633486+0000","last_clean":"2024-04-06T02:16:25.633486+0000","last_became_active":"2024-04-06T02:16:13.704043+0000","last_became_peered":"2024-04-06T02:16:13.704043+0000","last_unstale":"2024-04-06T02:16:25.633486+0000","last_undegraded":"2024-04-06T02:16:25.633486+0000","last_fullsized":"2024-04-06T02:16:25.633486+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T02:14:13.805654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T02:14:13.805654+0000","last_clean_scrub_stamp":"2024-04-06T02:14:13.805654+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:30.706142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.63100000000000001}]},{"osd":3,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.40100000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.505}]},{"osd":6,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52300000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Apr 6 02:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.439},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 6 02:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.27100000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38800000000000001,"15min":0.36699999999999999},"min":{"1min":0.26800000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Apr 6 02:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.29899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.24299999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.441},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.317},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40200000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79400000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.37},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Apr 6 02:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 02:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 02:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35099999999999998,"15min":0.33100000000000002},"min":{"1min":0.26400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61199999999999999,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Apr 6 02:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39500000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.248,"15min":0.248},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.502}]},{"osd":3,"last update":"Sat Apr 6 02:16:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.46600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Apr 6 02:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73999999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84699999999999998,"5min":0.871,"15min":0.871},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 6 02:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70399999999999996}]},{"osd":6,"last update":"Sat Apr 6 02:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.622},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Sat Apr 6 02:17:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38500000000000001,"15min":0.38600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 02:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58099999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Apr 6 02:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.29499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.26400000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.38800000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77100000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Apr 6 02:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.41499999999999998}]},{"osd":6,"last update":"Sat Apr 6 02:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44600000000000001}]},{"osd":7,"last update":"Sat Apr 6 02:17:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48399999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.255},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39500000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.314,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57699999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Apr 6 02:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.34799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.46600000000000003},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.315}]},{"osd":4,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77100000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.49299999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.76500000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Apr 6 02:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91100000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":0.73599999999999999,"15min":0.64600000000000002},"min":{"1min":0.48099999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":14.262,"5min":14.262,"15min":14.262},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Apr 6 02:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 02:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65800000000000003,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.65300000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56499999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.75,"15min":0.75},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64100000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.5}]},{"osd":3,"last update":"Sat Apr 6 02:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53900000000000003,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.504,"5min":1.276,"15min":1.4039999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.65200000000000002,"5min":18.692,"15min":18.692},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Apr 6 02:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.38,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.44,"15min":0.42599999999999999},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33400000000000002}]},{"osd":6,"last update":"Sat Apr 6 02:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35699999999999998}]},{"osd":7,"last update":"Sat Apr 6 02:17:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.38100000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":236988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.7610000000000001,"5min":5.7610000000000001,"15min":5.7610000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.42299999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69199999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.64600000000000002}]},{"osd":2,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70699999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.73899999999999999}]},{"osd":3,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.69399999999999995,"5min":0.753,"15min":0.753},"last":0.752},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 6 02:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56799999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.41499999999999998},"min":{"1min":0.32300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64700000000000002,"5min":0.877,"15min":0.877},"last":0.55100000000000005}]},{"osd":6,"last update":"Sat Apr 6 02:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Apr 6 02:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T02:17:41.574 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T02:17:41.574 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T02:17:41.575 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-06T02:17:41.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-06T02:17:41.793 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T02:17:41.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:41 smithi104 ceph-mon[25739]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:41.914 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:41 smithi017 ceph-mon[29567]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:41.914 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:41 smithi017 ceph-mon[24844]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:17:42.813 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:42 smithi017 ceph-mon[29567]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:42.814 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:42 smithi017 ceph-mon[24844]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:42.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:42 smithi104 ceph-mon[25739]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:43.564 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-04-06T02:17:43.565 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T02:17:43.894 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:43 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2405743878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T02:17:43.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:43 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2405743878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T02:17:43.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:43 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2405743878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T02:17:44.203 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-06T02:17:44.203 INFO:tasks.cephadm:Setup complete, yielding 2024-04-06T02:17:44.203 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:17:44.215 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-04-06T02:17:44.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-06T02:17:44.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:44 smithi017 ceph-mon[24844]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:44.849 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:44 smithi017 ceph-mon[29567]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:44 smithi104 ceph-mon[25739]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:45.598 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[47617]: level=info ts=2024-04-06T02:17:45.341Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001712972s 2024-04-06T02:17:46.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:46 smithi104 ceph-mon[25739]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:46.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:46 smithi017 ceph-mon[29567]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:46.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:46] "GET /metrics HTTP/1.1" 200 191486 "" "Prometheus/2.33.4" 2024-04-06T02:17:46.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:46 smithi017 ceph-mon[24844]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "2a7df0bb-cf8d-497f-8405-2f2a1449b7bb", 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "r", 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "current_period": "0aa36e37-f003-4746-8c2b-af43a4c07a25", 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2024-04-06T02:17:47.667 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:17:47.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:47 smithi104 ceph-mon[25739]: osdmap e60: 8 total, 8 up, 8 in 2024-04-06T02:17:47.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:47 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T02:17:47.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:47 smithi017 ceph-mon[29567]: osdmap e60: 8 total, 8 up, 8 in 2024-04-06T02:17:47.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:47 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T02:17:47.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:47 smithi017 ceph-mon[24844]: osdmap e60: 8 total, 8 up, 8 in 2024-04-06T02:17:47.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:47 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T02:17:48.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[24844]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[24844]: osdmap e61: 8 total, 8 up, 8 in 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[29567]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T02:17:48.904 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:48 smithi017 ceph-mon[29567]: osdmap e61: 8 total, 8 up, 8 in 2024-04-06T02:17:48.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:48 smithi104 ceph-mon[25739]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:48.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:48 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/180990585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T02:17:48.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:48 smithi104 ceph-mon[25739]: osdmap e61: 8 total, 8 up, 8 in 2024-04-06T02:17:49.895 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:49 smithi104 ceph-mon[25739]: osdmap e62: 8 total, 8 up, 8 in 2024-04-06T02:17:49.896 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:49 smithi104 podman[41804]: 2024-04-06 02:17:49.910328253 +0000 UTC m=+0.125363068 container create 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8, vendor=Red Hat, Inc., description=Ceph Grafana Container, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, distribution-scope=public, architecture=x86_64, build-date=2022-03-28T10:36:18.413762) 2024-04-06T02:17:49.911 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:49 smithi017 ceph-mon[29567]: osdmap e62: 8 total, 8 up, 8 in 2024-04-06T02:17:49.911 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:49 smithi017 ceph-mon[24844]: osdmap e62: 8 total, 8 up, 8 in 2024-04-06T02:17:50.087 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7cd59a28-0c1e-42e3-a100-b619d7e1fcf2", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [], 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [], 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "", 2024-04-06T02:17:50.088 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "2a7df0bb-cf8d-497f-8405-2f2a1449b7bb", 2024-04-06T02:17:50.089 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-04-06T02:17:50.089 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-04-06T02:17:50.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:17:50.089 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:17:50.146 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:17:50.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 podman[41804]: 2024-04-06 02:17:50.129395406 +0000 UTC m=+0.344430216 container init 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, distribution-scope=public, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , name=ubi8) 2024-04-06T02:17:50.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 podman[41804]: 2024-04-06 02:17:50.13237188 +0000 UTC m=+0.347406690 container start 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, description=Ceph Grafana Container, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2) 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-06T02:17:50.397 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="App mode production" logger=settings 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-06T02:17:50.398 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-06T02:17:50.399 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-06T02:17:50.400 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-06T02:17:50.401 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 bash[41804]: 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-06T02:17:50.402 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-06T02:17:50.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-06T02:17:50.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-06T02:17:50.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-06T02:17:50.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-06T02:17:50.405 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-06T02:17:50.406 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-06T02:17:50.407 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-06T02:17:50.408 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-06T02:17:50.409 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-06T02:17:50.410 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-06T02:17:50.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-06T02:17:50.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-06T02:17:50.413 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-06T02:17:50.646 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:50.646 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:50.646 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:17:50.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:17:50.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:17:50.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:50.647 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-06T02:17:50.647 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-06T02:17:50.647 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-06T02:17:50.648 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-06T02:17:50.649 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-06T02:17:50.650 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-06T02:17:50.651 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-06T02:17:50.652 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-06T02:17:50.653 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-06T02:17:50.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-06T02:17:50.655 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-06T02:17:50.656 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-06T02:17:50.657 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-06T02:17:50.658 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-06T02:17:50.659 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-06T02:17:50.660 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-06T02:17:50.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-06T02:17:50.662 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-06T02:17:50.662 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-06T02:17:50.662 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-06T02:17:50.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-06T02:17:50.914 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=544.783998ms 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-06T02:17:50.915 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-06T02:17:50.916 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-06T02:17:50.916 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-06T02:17:50.916 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-06T02:17:50.916 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-06T02:17:50.916 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:17:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:17:50+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:50.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:17:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:17:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:17:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:17:50.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:50 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:52.835 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:52 smithi017 ceph-mon[29567]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-06T02:17:52.836 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:52 smithi017 ceph-mon[24844]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-06T02:17:52.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:52 smithi104 ceph-mon[25739]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "99a3488c-1771-49a1-8bf6-cce9031e2068", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "z.rgw.meta:root", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "z.rgw.control", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "z.rgw.log", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-06T02:17:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "z.rgw.otp", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2024-04-06T02:17:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "2a7df0bb-cf8d-497f-8405-2f2a1449b7bb", 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-06T02:17:53.008 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:17:53.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-06T02:17:54.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:54 smithi017 ceph-mon[29567]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-06T02:17:54.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:54 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:54.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:54 smithi017 ceph-mon[24844]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-06T02:17:54.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:54 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:55.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:54 smithi104 ceph-mon[25739]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-06T02:17:55.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:54 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:56 smithi104 ceph-mon[25739]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-06T02:17:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:56 smithi104 ceph-mon[25739]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:17:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:56 smithi104 ceph-mon[25739]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:17:56.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[29567]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[29567]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[29567]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[24844]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[24844]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:17:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-mon[24844]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:17:56.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:17:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:17:56] "GET /metrics HTTP/1.1" 200 192939 "" "Prometheus/2.33.4" 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[24844]: osdmap e63: 8 total, 8 up, 8 in 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[24844]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[29567]: osdmap e63: 8 total, 8 up, 8 in 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:58 smithi017 ceph-mon[29567]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-06T02:17:58.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:58 smithi104 ceph-mon[25739]: osdmap e63: 8 total, 8 up, 8 in 2024-04-06T02:17:58.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:58 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:58 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-06T02:17:58.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:58 smithi104 ceph-mon[25739]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-06T02:17:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:59 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-06T02:17:59.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:17:59 smithi017 ceph-mon[24844]: osdmap e64: 8 total, 8 up, 8 in 2024-04-06T02:17:59.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:59 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-06T02:17:59.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:17:59 smithi017 ceph-mon[29567]: osdmap e64: 8 total, 8 up, 8 in 2024-04-06T02:17:59.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:59 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-06T02:17:59.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:17:59 smithi104 ceph-mon[25739]: osdmap e64: 8 total, 8 up, 8 in 2024-04-06T02:17:59.780 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:59 smithi017 systemd[1]: Stopping Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:17:59.780 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:59 smithi017 bash[49322]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-06T02:17:59.780 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[47617]: level=info ts=2024-04-06T02:17:59.774Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:18:00.076 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:17:59 smithi017 podman[49336]: 2024-04-06 02:17:59.794919209 +0000 UTC m=+0.129830868 container died b22a0310a41972cdb1fc47e74104743f2e7314769bc7f05f921431b803e28c79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:00.385 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49336]: 2024-04-06 02:18:00.207417576 +0000 UTC m=+0.542329239 container cleanup b22a0310a41972cdb1fc47e74104743f2e7314769bc7f05f921431b803e28c79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:00.385 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 bash[49336]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T02:18:00.385 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 bash[49362]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-06T02:18:00.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:18:00.697 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49350]: 2024-04-06 02:18:00.400302409 +0000 UTC m=+0.601841888 container remove b22a0310a41972cdb1fc47e74104743f2e7314769bc7f05f921431b803e28c79 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:00.697 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49350]: 2024-04-06 02:18:00.484558476 +0000 UTC m=+0.686097910 volume remove 1c7523e3ffcec3acafee0c93e66ce8137a5a52b0acbbacb7d1b597cdcf4baaee 2024-04-06T02:18:00.697 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-06T02:18:00.697 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 systemd[1]: Stopped Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:18:00.698 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 systemd[1]: Starting Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: osdmap e65: 8 total, 8 up, 8 in 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-06T02:18:00.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[29567]: osdmap e66: 8 total, 8 up, 8 in 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: osdmap e65: 8 total, 8 up, 8 in 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-06T02:18:00.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:00 smithi017 ceph-mon[24844]: osdmap e66: 8 total, 8 up, 8 in 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: osdmap e65: 8 total, 8 up, 8 in 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4127111971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-06T02:18:00.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:00 smithi104 ceph-mon[25739]: osdmap e66: 8 total, 8 up, 8 in 2024-04-06T02:18:00.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49430]: 2024-04-06 02:18:00.614977703 +0000 UTC m=+0.017690328 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-06T02:18:00.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49430]: 2024-04-06 02:18:00.744034199 +0000 UTC m=+0.146746853 volume create 0b5bb52e70527c9ac0475790951d274e91e24817fb4edb9836ca97624d25f63d 2024-04-06T02:18:00.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:00 smithi017 podman[49430]: 2024-04-06 02:18:00.844377219 +0000 UTC m=+0.247089830 container create e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:01.490 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 podman[49430]: 2024-04-06 02:18:01.105842781 +0000 UTC m=+0.508555409 container init e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:01.490 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 podman[49430]: 2024-04-06 02:18:01.108446007 +0000 UTC m=+0.511158619 container start e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:01.490 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.117Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-06T02:18:01.490 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.118Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.119Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.120Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.145Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.145Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.147Z caller=main.go:518 msg=Listening address=:9093 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:01.147Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 bash[49430]: e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c 2024-04-06T02:18:01.491 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:01 smithi017 systemd[1]: Started Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: osdmap e67: 8 total, 8 up, 8 in 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[24844]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: osdmap e67: 8 total, 8 up, 8 in 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:02 smithi017 ceph-mon[29567]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:02.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:02.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:18:02.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:18:02.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: osdmap e67: 8 total, 8 up, 8 in 2024-04-06T02:18:02.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T02:18:02.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:02 smithi104 ceph-mon[25739]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:03.419 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:03.121Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000520213s 2024-04-06T02:18:03.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:03 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-06T02:18:03.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:03 smithi104 ceph-mon[25739]: osdmap e68: 8 total, 8 up, 8 in 2024-04-06T02:18:03.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:03 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:03.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:03 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:03.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-06T02:18:03.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[29567]: osdmap e68: 8 total, 8 up, 8 in 2024-04-06T02:18:03.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:03.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:03.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-06T02:18:03.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[24844]: osdmap e68: 8 total, 8 up, 8 in 2024-04-06T02:18:03.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:03.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:03 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T02:18:04.485 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "4a747a63-2b1a-4844-9acd-b5c0ee9518b8", 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "0aa36e37-f003-4746-8c2b-af43a4c07a25", 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "4a747a63-2b1a-4844-9acd-b5c0ee9518b8", 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7cd59a28-0c1e-42e3-a100-b619d7e1fcf2", 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-04-06T02:18:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "99a3488c-1771-49a1-8bf6-cce9031e2068", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "99a3488c-1771-49a1-8bf6-cce9031e2068", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": "false", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": "false", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": "false", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2024-04-06T02:18:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": "true", 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "" 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:18:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "2a7df0bb-cf8d-497f-8405-2f2a1449b7bb", 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "99a3488c-1771-49a1-8bf6-cce9031e2068", 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 294895890 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "7cd59a28-0c1e-42e3-a100-b619d7e1fcf2", 2024-04-06T02:18:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "99a3488c-1771-49a1-8bf6-cce9031e2068", 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-04-06T02:18:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-04-06T02:18:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "2a7df0bb-cf8d-497f-8405-2f2a1449b7bb", 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "r", 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2024-04-06T02:18:04.492 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:18:04.564 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 systemd[1]: Stopping Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:18:04.565 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 bash[42498]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:18:04.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T02:18:04.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-mon[25739]: osdmap e69: 8 total, 8 up, 8 in 2024-04-06T02:18:04.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-mon[25739]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[29567]: osdmap e69: 8 total, 8 up, 8 in 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[29567]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[24844]: osdmap e69: 8 total, 8 up, 8 in 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/384924559' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-06T02:18:04.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:04 smithi017 ceph-mon[24844]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T02:18:04.882 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-06T02:18:04.883 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T02:18:04.883 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T02:18:04.883 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.579Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-06T02:18:04.883 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[40803]: ts=2024-04-06T02:18:04.580Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-06T02:18:04.883 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 podman[42511]: 2024-04-06 02:18:04.589466328 +0000 UTC m=+0.137128347 container died 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 podman[42511]: 2024-04-06 02:18:04.995172648 +0000 UTC m=+0.542834673 container cleanup 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:04 smithi104 bash[42511]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T02:18:05.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 bash[42539]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:18:05.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-06T02:18:05.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-06T02:18:05.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-mon[25739]: osdmap e70: 8 total, 8 up, 8 in 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 podman[42525]: 2024-04-06 02:18:05.221603718 +0000 UTC m=+0.634488143 container remove 9d878054731c5c9637254cfcdc6fe3fe7847b607471014b5c10c8ccbeb01ea1c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 systemd[1]: Stopped Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 systemd[1]: Starting Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 podman[42607]: 2024-04-06 02:18:05.455690192 +0000 UTC m=+0.114195641 container create 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.550 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 podman[42607]: 2024-04-06 02:18:05.360325519 +0000 UTC m=+0.018830981 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-06T02:18:05.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:05 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-06T02:18:05.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:05 smithi017 ceph-mon[24844]: osdmap e70: 8 total, 8 up, 8 in 2024-04-06T02:18:05.715 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:05 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-06T02:18:05.715 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:05 smithi017 ceph-mon[29567]: osdmap e70: 8 total, 8 up, 8 in 2024-04-06T02:18:05.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 podman[42607]: 2024-04-06 02:18:05.66620811 +0000 UTC m=+0.324713559 container init 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 podman[42607]: 2024-04-06 02:18:05.668904787 +0000 UTC m=+0.327410237 container start 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi104 (none))" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.691Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.693Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.693Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.695Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.696Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.696Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.805µs 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.696Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T02:18:05.914 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 bash[42607]: 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 systemd[1]: Started Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.888Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.888Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.888Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.087µs wal_replay_duration=191.987935ms total_replay_duration=192.027493ms 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.891Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.891Z caller=main.go:947 level=info msg="TSDB started" 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.891Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.905Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.435972ms db_storage=699ns remote_storage=1.13µs web_handler=380ns query_engine=1.245µs scrape=357.394µs scrape_sd=24.751µs notify=23.131µs notify_sd=7.727µs rules=13.786885ms 2024-04-06T02:18:05.915 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:18:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:18:05.905Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-06T02:18:07.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 495 B/s wr, 7 op/s 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:07.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:07.195 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled rgw.foo update... 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 495 B/s wr, 7 op/s 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 495 B/s wr, 7 op/s 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-06T02:18:07.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:07.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:07.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:07.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:08.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[29567]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:08.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[29567]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:08.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[24844]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:08.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[24844]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:08.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:08.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[24844]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-06T02:18:08.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch apply rgw smpl' 2024-04-06T02:18:08.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:08.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:08 smithi017 ceph-mon[29567]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-06T02:18:08.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:08 smithi104 ceph-mon[25739]: from='client.24580 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:08.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:08 smithi104 ceph-mon[25739]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:08.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:08 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:08.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:08 smithi104 ceph-mon[25739]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-06T02:18:10.253 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled rgw.smpl update... 2024-04-06T02:18:10.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:10 smithi104 ceph-mon[25739]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 1005 B/s wr, 2 op/s 2024-04-06T02:18:10.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:10 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:10.913 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:10 smithi017 ceph-mon[29567]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 1005 B/s wr, 2 op/s 2024-04-06T02:18:10.913 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:10 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:10.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:10 smithi017 ceph-mon[24844]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 1005 B/s wr, 2 op/s 2024-04-06T02:18:10.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:10 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:10.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph osd pool create foo' 2024-04-06T02:18:11.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:11 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:18:11.123Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002384939s 2024-04-06T02:18:11.627 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:11 smithi017 ceph-mon[29567]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:11.627 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:11 smithi017 ceph-mon[29567]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:11.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:11 smithi017 ceph-mon[24844]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:11.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:11 smithi017 ceph-mon[24844]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:11.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:11 smithi104 ceph-mon[25739]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:11.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:11 smithi104 ceph-mon[25739]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:12.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:12 smithi104 ceph-mon[25739]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 881 B/s wr, 2 op/s 2024-04-06T02:18:12.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:12 smithi017 ceph-mon[29567]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 881 B/s wr, 2 op/s 2024-04-06T02:18:12.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:12 smithi017 ceph-mon[24844]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 881 B/s wr, 2 op/s 2024-04-06T02:18:13.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:13 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:13.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:13 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:13.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:13 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.612 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'foo' created 2024-04-06T02:18:14.890 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2605938796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:14.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2605938796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:14 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2605938796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:14.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:14.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:14.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:14 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:15.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'rbd pool init foo' 2024-04-06T02:18:15.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:15 smithi104 ceph-mon[25739]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:15.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:15 smithi104 ceph-mon[25739]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:18:15.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:15 smithi104 ceph-mon[25739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-06T02:18:15.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:15 smithi104 ceph-mon[25739]: osdmap e71: 8 total, 8 up, 8 in 2024-04-06T02:18:15.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[29567]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[29567]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[29567]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[29567]: osdmap e71: 8 total, 8 up, 8 in 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[24844]: Saving service rgw.foo spec with placement count:2 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[24844]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[24844]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-06T02:18:15.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:15 smithi017 ceph-mon[24844]: osdmap e71: 8 total, 8 up, 8 in 2024-04-06T02:18:16.714 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:16 smithi017 ceph-mon[29567]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:16.714 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:16 smithi017 ceph-mon[29567]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T02:18:16.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:16 smithi017 ceph-mon[24844]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:16.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:16 smithi017 ceph-mon[24844]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T02:18:16.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:16 smithi104 ceph-mon[25739]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-06T02:18:16.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:16 smithi104 ceph-mon[25739]: osdmap e72: 8 total, 8 up, 8 in 2024-04-06T02:18:16.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:18:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:16] "GET /metrics HTTP/1.1" 200 196822 "" "Prometheus/2.33.4" 2024-04-06T02:18:17.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:17 smithi017 ceph-mon[29567]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T02:18:17.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:17 smithi017 ceph-mon[24844]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T02:18:17.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:17 smithi104 ceph-mon[25739]: osdmap e73: 8 total, 8 up, 8 in 2024-04-06T02:18:18.855 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:18 smithi017 ceph-mon[29567]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:18.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:18 smithi017 ceph-mon[24844]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:18 smithi104 ceph-mon[25739]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:19.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-06T02:18:19.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-06T02:18:19.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: osdmap e74: 8 total, 8 up, 8 in 2024-04-06T02:18:19.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:19.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:19 smithi104 ceph-mon[25739]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:18:19.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: osdmap e74: 8 total, 8 up, 8 in 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[29567]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3347976868' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-06T02:18:19.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: osdmap e74: 8 total, 8 up, 8 in 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:19.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:19 smithi017 ceph-mon[24844]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:18:20.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:18:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:18:20.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:20 smithi104 ceph-mon[25739]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:20.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:20 smithi104 ceph-mon[25739]: osdmap e75: 8 total, 8 up, 8 in 2024-04-06T02:18:20.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:20 smithi017 ceph-mon[29567]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:20.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:20 smithi017 ceph-mon[29567]: osdmap e75: 8 total, 8 up, 8 in 2024-04-06T02:18:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:20 smithi017 ceph-mon[24844]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:20 smithi017 ceph-mon[24844]: osdmap e75: 8 total, 8 up, 8 in 2024-04-06T02:18:21.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-06T02:18:21.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: osdmap e76: 8 total, 8 up, 8 in 2024-04-06T02:18:21.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: osdmap e76: 8 total, 8 up, 8 in 2024-04-06T02:18:21.991 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:21.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:21.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:21.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:21 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:22.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: osdmap e76: 8 total, 8 up, 8 in 2024-04-06T02:18:22.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:22.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:22.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:22.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:22.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:22.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:21 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[29567]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[29567]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[29567]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 7.3 KiB/s wr, 547 op/s 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[24844]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[24844]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:18:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:22 smithi017 ceph-mon[24844]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 7.3 KiB/s wr, 547 op/s 2024-04-06T02:18:23.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:22 smithi104 ceph-mon[25739]: Saving service rgw.smpl spec with placement count:2 2024-04-06T02:18:23.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:22 smithi104 ceph-mon[25739]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:18:23.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:22 smithi104 ceph-mon[25739]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 7.3 KiB/s wr, 547 op/s 2024-04-06T02:18:23.842 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled iscsi.foo update... 2024-04-06T02:18:25.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'sleep 120' 2024-04-06T02:18:25.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[29567]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:25.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[29567]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:25.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[29567]: Saving service iscsi.foo spec with placement count:1 2024-04-06T02:18:25.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[29567]: osdmap e77: 8 total, 8 up, 8 in 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[24844]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[24844]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[24844]: Saving service iscsi.foo spec with placement count:1 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:25.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:24 smithi017 ceph-mon[24844]: osdmap e77: 8 total, 8 up, 8 in 2024-04-06T02:18:25.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:24 smithi104 ceph-mon[25739]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:25.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:24 smithi104 ceph-mon[25739]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:18:25.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:24 smithi104 ceph-mon[25739]: Saving service iscsi.foo spec with placement count:1 2024-04-06T02:18:25.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:24 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:25.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:24 smithi104 ceph-mon[25739]: osdmap e77: 8 total, 8 up, 8 in 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:26.112 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:26.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:26.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:18:26.973 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:26] "GET /metrics HTTP/1.1" 200 207240 "" "Prometheus/2.33.4" 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[29567]: osdmap e78: 8 total, 8 up, 8 in 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-04-06T02:18:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-06T02:18:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-06T02:18:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:26 smithi017 ceph-mon[24844]: osdmap e78: 8 total, 8 up, 8 in 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 247 KiB/s rd, 6.0 KiB/s wr, 450 op/s 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-06T02:18:27.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:26 smithi104 ceph-mon[25739]: osdmap e78: 8 total, 8 up, 8 in 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: osdmap e79: 8 total, 8 up, 8 in 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:27 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:28.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: osdmap e79: 8 total, 8 up, 8 in 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: osdmap e79: 8 total, 8 up, 8 in 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:27 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:29.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:28 smithi104 ceph-mon[25739]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:29.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:28 smithi104 ceph-mon[25739]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:18:29.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:28 smithi017 ceph-mon[29567]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:29.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:28 smithi017 ceph-mon[29567]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:18:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:28 smithi017 ceph-mon[24844]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:28 smithi017 ceph-mon[24844]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:18:30.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:18:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:31.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:31 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 90 op/s 2024-04-06T02:18:31.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:31.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:31 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:33.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:33 smithi104 ceph-mon[25739]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 313 KiB/s rd, 292 B/s wr, 480 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:33.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:33 smithi017 ceph-mon[29567]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 313 KiB/s rd, 292 B/s wr, 480 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:33.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:33 smithi017 ceph-mon[24844]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 313 KiB/s rd, 292 B/s wr, 480 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:33.926 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:18:33.926 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:18:34.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 274 KiB/s rd, 255 B/s wr, 421 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: Checking pool "foo" exists for service iscsi.foo 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:34.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[29567]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 274 KiB/s rd, 255 B/s wr, 421 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: Checking pool "foo" exists for service iscsi.foo 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:34.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:34 smithi017 ceph-mon[24844]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 274 KiB/s rd, 255 B/s wr, 421 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: Checking pool "foo" exists for service iscsi.foo 2024-04-06T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:18:34.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T02:18:34.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:34.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:34 smithi104 ceph-mon[25739]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:18:36.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:36 smithi017 ceph-mon[24844]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 210 B/s wr, 347 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:36.740 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:18:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:36] "GET /metrics HTTP/1.1" 200 207240 "" "Prometheus/2.33.4" 2024-04-06T02:18:36.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:36 smithi017 ceph-mon[29567]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 210 B/s wr, 347 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:36.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:36 smithi104 ceph-mon[25739]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 210 B/s wr, 347 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/973192022' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:18:37.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]: dispatch 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/973192022' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:18:37.893 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:37 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]: dispatch 2024-04-06T02:18:37.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/973192022' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:18:37.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:37 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]: dispatch 2024-04-06T02:18:39.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:38 smithi104 ceph-mon[25739]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 382 B/s wr, 292 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:39.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:38 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]': finished 2024-04-06T02:18:39.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:38 smithi104 ceph-mon[25739]: osdmap e80: 8 total, 8 up, 8 in 2024-04-06T02:18:39.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:38 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]: dispatch 2024-04-06T02:18:39.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[29567]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 382 B/s wr, 292 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:39.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]': finished 2024-04-06T02:18:39.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[29567]: osdmap e80: 8 total, 8 up, 8 in 2024-04-06T02:18:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]: dispatch 2024-04-06T02:18:39.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[24844]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 382 B/s wr, 292 op/s; 0 B/s, 0 objects/s recovering 2024-04-06T02:18:39.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1940725479' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438695043"}]': finished 2024-04-06T02:18:39.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[24844]: osdmap e80: 8 total, 8 up, 8 in 2024-04-06T02:18:39.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:38 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]: dispatch 2024-04-06T02:18:39.976 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:39 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]': finished 2024-04-06T02:18:39.976 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:39 smithi104 ceph-mon[25739]: osdmap e81: 8 total, 8 up, 8 in 2024-04-06T02:18:39.976 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:39 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3183411676' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:39.976 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:39 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]': finished 2024-04-06T02:18:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[24844]: osdmap e81: 8 total, 8 up, 8 in 2024-04-06T02:18:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3183411676' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:40.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:40.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1272226202' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/397925205"}]': finished 2024-04-06T02:18:40.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[29567]: osdmap e81: 8 total, 8 up, 8 in 2024-04-06T02:18:40.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3183411676' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:40.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:39 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]: dispatch 2024-04-06T02:18:40.256 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]': finished 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: osdmap e82: 8 total, 8 up, 8 in 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4254526360' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:41.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:40 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]': finished 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: osdmap e82: 8 total, 8 up, 8 in 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4254526360' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/551721499"}]': finished 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: osdmap e82: 8 total, 8 up, 8 in 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4254526360' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:18:41.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:40 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]': finished 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: osdmap e83: 8 total, 8 up, 8 in 2024-04-06T02:18:42.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3310835223' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:41 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]': finished 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: osdmap e83: 8 total, 8 up, 8 in 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3310835223' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2756296616"}]': finished 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: osdmap e83: 8 total, 8 up, 8 in 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3310835223' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:41 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]: dispatch 2024-04-06T02:18:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[24844]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]': finished 2024-04-06T02:18:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[24844]: osdmap e84: 8 total, 8 up, 8 in 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/392741799' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[29567]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]': finished 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[29567]: osdmap e84: 8 total, 8 up, 8 in 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/392741799' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:42.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:43.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:42 smithi104 ceph-mon[25739]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:43.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/397925205"}]': finished 2024-04-06T02:18:43.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:42 smithi104 ceph-mon[25739]: osdmap e84: 8 total, 8 up, 8 in 2024-04-06T02:18:43.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/392741799' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:43.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]: dispatch 2024-04-06T02:18:43.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:18:43.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:18:43.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:18:43.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:18:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:43 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]': finished 2024-04-06T02:18:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:43 smithi104 ceph-mon[25739]: osdmap e85: 8 total, 8 up, 8 in 2024-04-06T02:18:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:43 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/866057549' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:43 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:43 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]': finished 2024-04-06T02:18:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[29567]: osdmap e85: 8 total, 8 up, 8 in 2024-04-06T02:18:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/866057549' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2917214294"}]': finished 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[24844]: osdmap e85: 8 total, 8 up, 8 in 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/866057549' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]: dispatch 2024-04-06T02:18:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:43 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]': finished 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: osdmap e86: 8 total, 8 up, 8 in 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/668329438' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:44 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]': finished 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: osdmap e86: 8 total, 8 up, 8 in 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/668329438' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 255 B/s wr, 28 op/s 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4184866883"}]': finished 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: osdmap e86: 8 total, 8 up, 8 in 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: Checking dashboard <-> RGW credentials 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/668329438' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]: dispatch 2024-04-06T02:18:45.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:44 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:18:46.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:46 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]': finished 2024-04-06T02:18:46.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:46 smithi104 ceph-mon[25739]: osdmap e87: 8 total, 8 up, 8 in 2024-04-06T02:18:46.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:46 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2143470559' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:46 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]': finished 2024-04-06T02:18:46.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[29567]: osdmap e87: 8 total, 8 up, 8 in 2024-04-06T02:18:46.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2143470559' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4025515538"}]': finished 2024-04-06T02:18:46.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[24844]: osdmap e87: 8 total, 8 up, 8 in 2024-04-06T02:18:46.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2143470559' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]: dispatch 2024-04-06T02:18:46.985 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:18:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:46] "GET /metrics HTTP/1.1" 200 214970 "" "Prometheus/2.33.4" 2024-04-06T02:18:47.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[29567]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:47.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]': finished 2024-04-06T02:18:47.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[29567]: osdmap e88: 8 total, 8 up, 8 in 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1743835647' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[24844]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]': finished 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[24844]: osdmap e88: 8 total, 8 up, 8 in 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1743835647' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:47.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:47 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:47.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:47 smithi104 ceph-mon[25739]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:18:47.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:47 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2438202178"}]': finished 2024-04-06T02:18:47.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:47 smithi104 ceph-mon[25739]: osdmap e88: 8 total, 8 up, 8 in 2024-04-06T02:18:47.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:47 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1743835647' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:47.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:47 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]: dispatch 2024-04-06T02:18:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:48 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:48 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]': finished 2024-04-06T02:18:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:48 smithi104 ceph-mon[25739]: osdmap e89: 8 total, 8 up, 8 in 2024-04-06T02:18:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:48 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/565795169' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:48.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:48 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]': finished 2024-04-06T02:18:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[24844]: osdmap e89: 8 total, 8 up, 8 in 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/565795169' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1418949283"}]': finished 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[29567]: osdmap e89: 8 total, 8 up, 8 in 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/565795169' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:48.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:48 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]: dispatch 2024-04-06T02:18:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:49 smithi104 ceph-mon[25739]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:49 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]': finished 2024-04-06T02:18:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:49 smithi104 ceph-mon[25739]: osdmap e90: 8 total, 8 up, 8 in 2024-04-06T02:18:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:49 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2986624659' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:49.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:49 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:49.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[29567]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]': finished 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[29567]: osdmap e90: 8 total, 8 up, 8 in 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2986624659' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[24844]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3265016438"}]': finished 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[24844]: osdmap e90: 8 total, 8 up, 8 in 2024-04-06T02:18:49.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2986624659' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:49.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:49 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]: dispatch 2024-04-06T02:18:50.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:18:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:18:50.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:50 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]': finished 2024-04-06T02:18:50.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:50 smithi104 ceph-mon[25739]: osdmap e91: 8 total, 8 up, 8 in 2024-04-06T02:18:50.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:50 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/106260249' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:50.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:50 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:50.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]': finished 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[29567]: osdmap e91: 8 total, 8 up, 8 in 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/106260249' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2138725502"}]': finished 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[24844]: osdmap e91: 8 total, 8 up, 8 in 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/106260249' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:50 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]: dispatch 2024-04-06T02:18:51.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:51 smithi104 ceph-mon[25739]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:51.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:51 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]': finished 2024-04-06T02:18:51.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:51 smithi104 ceph-mon[25739]: osdmap e92: 8 total, 8 up, 8 in 2024-04-06T02:18:51.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:51 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]: dispatch 2024-04-06T02:18:51.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[29567]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:51.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]': finished 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[29567]: osdmap e92: 8 total, 8 up, 8 in 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]: dispatch 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[24844]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2138725502"}]': finished 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[24844]: osdmap e92: 8 total, 8 up, 8 in 2024-04-06T02:18:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:51 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]: dispatch 2024-04-06T02:18:52.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:52 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]': finished 2024-04-06T02:18:52.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:52 smithi104 ceph-mon[25739]: osdmap e93: 8 total, 8 up, 8 in 2024-04-06T02:18:52.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:52 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1765952924' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:52 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:52 smithi104 ceph-mon[25739]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]': finished 2024-04-06T02:18:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[29567]: osdmap e93: 8 total, 8 up, 8 in 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1765952924' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[29567]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1038908032' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1295449500"}]': finished 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[24844]: osdmap e93: 8 total, 8 up, 8 in 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1765952924' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]: dispatch 2024-04-06T02:18:52.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:52 smithi017 ceph-mon[24844]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]': finished 2024-04-06T02:18:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[24844]: osdmap e94: 8 total, 8 up, 8 in 2024-04-06T02:18:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1706021121' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]': finished 2024-04-06T02:18:53.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[29567]: osdmap e94: 8 total, 8 up, 8 in 2024-04-06T02:18:53.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1706021121' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:53 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4184866883"}]': finished 2024-04-06T02:18:53.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:53 smithi104 ceph-mon[25739]: osdmap e94: 8 total, 8 up, 8 in 2024-04-06T02:18:53.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:53 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1706021121' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:53 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]: dispatch 2024-04-06T02:18:53.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:18:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:18:53.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:18:53.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:18:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:18:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:18:54.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]': finished 2024-04-06T02:18:54.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[29567]: osdmap e95: 8 total, 8 up, 8 in 2024-04-06T02:18:54.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[29567]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:54.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]': finished 2024-04-06T02:18:54.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[24844]: osdmap e95: 8 total, 8 up, 8 in 2024-04-06T02:18:54.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:54 smithi017 ceph-mon[24844]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:54.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:54 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3736517106"}]': finished 2024-04-06T02:18:54.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:54 smithi104 ceph-mon[25739]: osdmap e95: 8 total, 8 up, 8 in 2024-04-06T02:18:54.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:54 smithi104 ceph-mon[25739]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:18:56.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:56 smithi104 ceph-mon[25739]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-04-06T02:18:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:56 smithi017 ceph-mon[24844]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-04-06T02:18:56.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:56 smithi017 ceph-mon[29567]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-04-06T02:18:56.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:18:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:18:56] "GET /metrics HTTP/1.1" 200 214983 "" "Prometheus/2.33.4" 2024-04-06T02:18:57.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:57 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:57 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:57 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:18:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:18:58 smithi104 ceph-mon[25739]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:18:58.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:18:58 smithi017 ceph-mon[29567]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:18:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:18:58 smithi017 ceph-mon[24844]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:19:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:00.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:00 smithi104 ceph-mon[25739]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:19:00.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:00 smithi017 ceph-mon[29567]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:19:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:00 smithi017 ceph-mon[24844]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:19:02.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:02 smithi104 ceph-mon[25739]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:19:02.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:02 smithi017 ceph-mon[29567]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:19:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:02 smithi017 ceph-mon[24844]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:19:03.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:19:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:19:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:19:03.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:19:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:19:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:19:03.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:19:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:19:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:19:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:04 smithi104 ceph-mon[25739]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-06T02:19:04.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:04 smithi017 ceph-mon[29567]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-06T02:19:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:04 smithi017 ceph-mon[24844]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-06T02:19:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:06 smithi104 ceph-mon[25739]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:06.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:06 smithi017 ceph-mon[29567]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:06 smithi017 ceph-mon[24844]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:06.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:06] "GET /metrics HTTP/1.1" 200 214983 "" "Prometheus/2.33.4" 2024-04-06T02:19:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:07 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:07.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:07 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:07 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:08.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:08 smithi104 ceph-mon[25739]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:08.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:08 smithi017 ceph-mon[29567]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:08 smithi017 ceph-mon[24844]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:10.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:10 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:10.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:10 smithi104 ceph-mon[25739]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:10 smithi017 ceph-mon[24844]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:10.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:10 smithi017 ceph-mon[29567]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:12 smithi104 ceph-mon[25739]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:12 smithi017 ceph-mon[24844]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:12.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:12 smithi017 ceph-mon[29567]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:13.990 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:19:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:19:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:19:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:14 smithi104 ceph-mon[25739]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:14 smithi017 ceph-mon[24844]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:14.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:14 smithi017 ceph-mon[29567]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:16.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:16 smithi104 ceph-mon[25739]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:16.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:16 smithi017 ceph-mon[29567]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:16 smithi017 ceph-mon[24844]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:16.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:16] "GET /metrics HTTP/1.1" 200 214928 "" "Prometheus/2.33.4" 2024-04-06T02:19:17.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:17 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:17 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:17.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:17 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:18.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:18 smithi104 ceph-mon[25739]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:18.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:18 smithi017 ceph-mon[29567]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:18 smithi017 ceph-mon[24844]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:20.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:20.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:20 smithi104 ceph-mon[25739]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:20.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:20 smithi017 ceph-mon[29567]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:20 smithi017 ceph-mon[24844]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:22.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:22 smithi104 ceph-mon[25739]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:22 smithi017 ceph-mon[29567]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:22 smithi017 ceph-mon[24844]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:24.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:24 smithi104 ceph-mon[25739]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:24 smithi017 ceph-mon[29567]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:24 smithi017 ceph-mon[24844]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:25.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:19:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:19:25.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:19:26.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:26 smithi104 ceph-mon[25739]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:26.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:19:26.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:26 smithi017 ceph-mon[29567]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:26.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:19:26.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:26 smithi017 ceph-mon[24844]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:26.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:19:26.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:26] "GET /metrics HTTP/1.1" 200 214883 "" "Prometheus/2.33.4" 2024-04-06T02:19:27.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:27 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:27.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:27 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:27 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:28.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:28 smithi104 ceph-mon[25739]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:28.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:28 smithi017 ceph-mon[29567]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:28 smithi017 ceph-mon[24844]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:30.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:30.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:30 smithi104 ceph-mon[25739]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:30.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:30 smithi017 ceph-mon[29567]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:30 smithi017 ceph-mon[24844]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:32.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:32 smithi104 ceph-mon[25739]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:32.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:32 smithi017 ceph-mon[29567]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:32 smithi017 ceph-mon[24844]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:34.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:34 smithi104 ceph-mon[25739]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:34.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:34 smithi017 ceph-mon[29567]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:34 smithi017 ceph-mon[24844]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:36.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:36 smithi104 ceph-mon[25739]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:36.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:36] "GET /metrics HTTP/1.1" 200 214883 "" "Prometheus/2.33.4" 2024-04-06T02:19:36.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:36 smithi017 ceph-mon[29567]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:36.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:36 smithi017 ceph-mon[24844]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:37.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:37 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:37.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:37 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:37 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:38.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:38 smithi104 ceph-mon[25739]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:38.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:38 smithi017 ceph-mon[29567]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:38 smithi017 ceph-mon[24844]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:40.413 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:40.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:40 smithi104 ceph-mon[25739]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:40.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:40 smithi017 ceph-mon[29567]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:40 smithi017 ceph-mon[24844]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:42 smithi104 ceph-mon[25739]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:42 smithi017 ceph-mon[29567]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:42 smithi017 ceph-mon[24844]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:44 smithi104 ceph-mon[25739]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:44 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:19:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:44 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:19:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:44 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:19:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[24844]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:19:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:19:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:19:44.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[29567]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:44.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:19:44.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:19:44.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:44 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:19:45.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:45 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:19:45.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:45 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:19:46.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:45 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:19:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:46 smithi017 ceph-mon[24844]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:46.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:46 smithi017 ceph-mon[29567]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:46.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:46] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-04-06T02:19:47.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:46 smithi104 ceph-mon[25739]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:47.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:47 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:47.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:47 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:47 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:48 smithi017 ceph-mon[24844]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:48 smithi017 ceph-mon[29567]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:48 smithi104 ceph-mon[25739]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:50.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:19:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:19:50.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:50 smithi104 ceph-mon[25739]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:50 smithi017 ceph-mon[24844]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:50.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:50 smithi017 ceph-mon[29567]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:52.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:52 smithi104 ceph-mon[25739]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:52 smithi017 ceph-mon[24844]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:52.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:52 smithi017 ceph-mon[29567]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:54.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:54 smithi104 ceph-mon[25739]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:54.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:54 smithi017 ceph-mon[29567]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:54 smithi017 ceph-mon[24844]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:56.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:56 smithi104 ceph-mon[25739]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:56 smithi017 ceph-mon[29567]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:56 smithi017 ceph-mon[24844]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:19:56.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:19:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:19:56] "GET /metrics HTTP/1.1" 200 214888 "" "Prometheus/2.33.4" 2024-04-06T02:19:57.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:57 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:57.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:57 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:57.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:57 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:19:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:19:58 smithi104 ceph-mon[25739]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:58.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:19:58 smithi017 ceph-mon[29567]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:19:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:19:58 smithi017 ceph-mon[24844]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:00.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:00 smithi104 ceph-mon[25739]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:00.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:00 smithi104 ceph-mon[25739]: overall HEALTH_OK 2024-04-06T02:20:00.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:00 smithi017 ceph-mon[29567]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:00.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:00 smithi017 ceph-mon[29567]: overall HEALTH_OK 2024-04-06T02:20:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:00 smithi017 ceph-mon[24844]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:00 smithi017 ceph-mon[24844]: overall HEALTH_OK 2024-04-06T02:20:02.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:02 smithi104 ceph-mon[25739]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:02.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:02 smithi017 ceph-mon[29567]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:02 smithi017 ceph-mon[24844]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:04 smithi104 ceph-mon[25739]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:04.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:04 smithi017 ceph-mon[29567]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:04 smithi017 ceph-mon[24844]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:06 smithi104 ceph-mon[25739]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:06.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:06 smithi017 ceph-mon[29567]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:06 smithi017 ceph-mon[24844]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:06.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:06] "GET /metrics HTTP/1.1" 200 214888 "" "Prometheus/2.33.4" 2024-04-06T02:20:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:07 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:07.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:07 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:07 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:08.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:08 smithi104 ceph-mon[25739]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:08.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:08 smithi017 ceph-mon[29567]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:08 smithi017 ceph-mon[24844]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:10.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:10 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:10.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:10 smithi104 ceph-mon[25739]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:10.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:10 smithi017 ceph-mon[29567]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:10 smithi017 ceph-mon[24844]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:12 smithi104 ceph-mon[25739]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:12.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:12 smithi017 ceph-mon[29567]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:12 smithi017 ceph-mon[24844]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:14 smithi104 ceph-mon[25739]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:14.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:14 smithi017 ceph-mon[29567]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:14 smithi017 ceph-mon[24844]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:16.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:16 smithi104 ceph-mon[25739]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:16.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:16 smithi017 ceph-mon[29567]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:16.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:16] "GET /metrics HTTP/1.1" 200 214887 "" "Prometheus/2.33.4" 2024-04-06T02:20:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:16 smithi017 ceph-mon[24844]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:17.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:17 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:17.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:17 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:17 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:18.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:18 smithi104 ceph-mon[25739]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:18.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:18 smithi017 ceph-mon[29567]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:18 smithi017 ceph-mon[24844]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:20.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:20.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:20 smithi104 ceph-mon[25739]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:20.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:20 smithi017 ceph-mon[29567]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:20 smithi017 ceph-mon[24844]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:22.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:22 smithi104 ceph-mon[25739]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:22 smithi017 ceph-mon[29567]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:22 smithi017 ceph-mon[24844]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:24.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:24 smithi104 ceph-mon[25739]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:24 smithi017 ceph-mon[24844]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:24 smithi017 ceph-mon[29567]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:25.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:20:25.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:20:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:20:26.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:26 smithi104 ceph-mon[25739]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:26.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:26 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:20:26.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:26 smithi017 ceph-mon[29567]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:26.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:26 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:20:26.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:26 smithi017 ceph-mon[24844]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:26.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:26 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:20:26.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:26] "GET /metrics HTTP/1.1" 200 214902 "" "Prometheus/2.33.4" 2024-04-06T02:20:27.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:27 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:27.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:27 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:27.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:27 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:27.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-06T02:20:28.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:28 smithi104 ceph-mon[25739]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:28.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:28 smithi017 ceph-mon[29567]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:28 smithi017 ceph-mon[24844]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:30.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:30 smithi017 ceph-mon[29567]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:30 smithi017 ceph-mon[24844]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:30.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-06T02:20:30.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:30 smithi104 ceph-mon[25739]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:32 smithi017 ceph-mon[24844]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:32.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:32 smithi017 ceph-mon[29567]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:32.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:32 smithi104 ceph-mon[25739]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:33.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-06T02:20:34.895 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:34 smithi017 ceph-mon[29567]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:34.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:34 smithi017 ceph-mon[24844]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:34.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:34 smithi104 ceph-mon[25739]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:35.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-06T02:20:36.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:36 smithi104 ceph-mon[25739]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:36.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:36 smithi017 ceph-mon[29567]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:36.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:36 smithi017 ceph-mon[24844]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:36.950 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:36] "GET /metrics HTTP/1.1" 200 214902 "" "Prometheus/2.33.4" 2024-04-06T02:20:37.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:37 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:37.931 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:37.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:37 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:37.932 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:37 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:38.572 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:20:38.581 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-04-06T02:20:38.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[24844]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:39.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:38 smithi017 ceph-mon[29567]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:39.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:39.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:39.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:39.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:20:39.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:20:39.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:20:39.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:20:39.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:38 smithi104 ceph-mon[25739]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:20:40.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:40.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:40 smithi017 ceph-mon[24844]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:40.841 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:40 smithi017 ceph-mon[29567]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:40.897 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:20:40.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:40 smithi104 ceph-mon[25739]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 117s ago 3m 17.9M - ba2b418f427c e252f7611c30 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 running (2m) 117s ago 2m 49.0M - 8.3.5 dad864ee21e9 40df0b2e5224 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (2m) 117s ago 2m 74.5M - 3.5 e1d6a67b021e fba4f4b5b4eb 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443 running (7m) 117s ago 7m 411M - 17.2.0 e1d6a67b021e 247dc63ae7b6 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (9m) 117s ago 9m 457M - 17.2.0 e1d6a67b021e a6a010afe065 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 117s ago 9m 48.8M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (7m) 117s ago 7m 39.2M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:20:41.781 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 117s ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 117s ago 3m 18.3M - 1dbe0e931976 db0c895136c8 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (3m) 117s ago 3m 16.4M - 1dbe0e931976 ad624bfcf398 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 117s ago 7m 50.2M 4017M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (6m) 117s ago 6m 51.6M 4017M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (6m) 117s ago 6m 45.7M 4017M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6m) 117s ago 6m 52.3M 4017M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (5m) 117s ago 5m 50.8M 4272M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (5m) 117s ago 5m 49.4M 4272M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (5m) 117s ago 5m 48.7M 4272M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (4m) 117s ago 4m 49.1M 4272M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (2m) 117s ago 3m 46.3M - 514e6a882f6e 8bde7995dc5e 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (2m) 117s ago 2m 69.9M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:20:41.782 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (2m) 117s ago 2m 69.2M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:20:41.783 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (2m) 117s ago 2m 72.1M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:20:41.783 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (2m) 117s ago 2m 73.0M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:20:41.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:41 smithi104 ceph-mon[25739]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:41.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:41 smithi017 ceph-mon[29567]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:41.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:41 smithi017 ceph-mon[24844]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.310 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:20:42.310 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:20:42.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:20:42.311 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "", 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 image" 2024-04-06T02:20:42.809 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:20:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:42 smithi104 ceph-mon[25739]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:42 smithi104 ceph-mon[25739]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:42 smithi104 ceph-mon[25739]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3088465273' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:20:42.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[24844]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[24844]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[24844]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3088465273' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[29567]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[29567]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[29567]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3088465273' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:20:43.275 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:20:43.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:43 smithi104 ceph-mon[25739]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:43.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:43 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3770460603' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:20:43.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:43 smithi017 ceph-mon[24844]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:43.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:43 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3770460603' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:20:43.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:43 smithi017 ceph-mon[29567]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:20:43.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:43 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3770460603' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:20:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:44 smithi104 ceph-mon[25739]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:44.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:44 smithi017 ceph-mon[29567]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:44 smithi017 ceph-mon[24844]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:46.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:46 smithi104 ceph-mon[25739]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:46.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:46 smithi017 ceph-mon[29567]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:46 smithi017 ceph-mon[24844]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:46.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:46] "GET /metrics HTTP/1.1" 200 214885 "" "Prometheus/2.33.4" 2024-04-06T02:20:47.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:47 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:47.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:47 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:47.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:47 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:48 smithi104 ceph-mon[25739]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:48.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:48 smithi017 ceph-mon[29567]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:48 smithi017 ceph-mon[24844]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:50.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:20:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:20:50.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:50 smithi104 ceph-mon[25739]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:50.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:50 smithi017 ceph-mon[29567]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:50 smithi017 ceph-mon[24844]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:52.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:52 smithi104 ceph-mon[25739]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:52.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:52 smithi017 ceph-mon[29567]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:52 smithi017 ceph-mon[24844]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:54.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:54 smithi104 ceph-mon[25739]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:54.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:54 smithi017 ceph-mon[29567]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:54.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:54 smithi017 ceph-mon[24844]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:56.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:56 smithi104 ceph-mon[25739]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:56 smithi017 ceph-mon[29567]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:56 smithi017 ceph-mon[24844]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:20:56.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:20:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:20:56] "GET /metrics HTTP/1.1" 200 214859 "" "Prometheus/2.33.4" 2024-04-06T02:20:57.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:57 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:57.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:57 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:57 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:20:58.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:20:58 smithi017 ceph-mon[24844]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:20:58 smithi104 ceph-mon[25739]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:20:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:20:58 smithi017 ceph-mon[29567]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:21:00.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:00 smithi104 ceph-mon[25739]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:00.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:00 smithi017 ceph-mon[29567]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:00.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:00 smithi017 ceph-mon[24844]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:02.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:02 smithi104 ceph-mon[25739]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:02.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:02 smithi017 ceph-mon[29567]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:02 smithi017 ceph-mon[24844]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:04 smithi104 ceph-mon[25739]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:04.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:04 smithi017 ceph-mon[29567]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:04.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:04 smithi017 ceph-mon[24844]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:06 smithi104 ceph-mon[25739]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:06.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:06 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[24844]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:06.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:06] "GET /metrics HTTP/1.1" 200 214859 "" "Prometheus/2.33.4" 2024-04-06T02:21:06.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[29567]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:06.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:06.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:06.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:06 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:07 smithi104 ceph-mon[25739]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T02:21:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:07 smithi104 ceph-mon[25739]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T02:21:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:07 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:07 smithi104 ceph-mon[25739]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi104 2024-04-06T02:21:07.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:07 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[24844]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[24844]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[24844]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi104 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:07.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[29567]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T02:21:07.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[29567]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T02:21:07.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:07.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[29567]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi104 2024-04-06T02:21:07.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:07 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:08.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:08 smithi104 ceph-mon[25739]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:08 smithi017 ceph-mon[29567]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:08 smithi017 ceph-mon[24844]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:10.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:10 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:21:10.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:10 smithi104 ceph-mon[25739]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:10 smithi017 ceph-mon[24844]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:10.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:10 smithi017 ceph-mon[29567]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:12 smithi104 ceph-mon[25739]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:12.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:12 smithi017 ceph-mon[29567]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:12 smithi017 ceph-mon[24844]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:13.616 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:21:13.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:21:13.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:14.220 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 2m ago 3m 17.9M - ba2b418f427c e252f7611c30 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 running (3m) 2m ago 3m 49.0M - 8.3.5 dad864ee21e9 40df0b2e5224 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (2m) 2m ago 2m 74.5M - 3.5 e1d6a67b021e fba4f4b5b4eb 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 247dc63ae7b6 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (9m) 2m ago 9m 457M - 17.2.0 e1d6a67b021e a6a010afe065 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 2m ago 9m 48.8M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 2m ago 4m 18.3M - 1dbe0e931976 db0c895136c8 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (4m) 2m ago 4m 16.4M - 1dbe0e931976 ad624bfcf398 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 2m ago 7m 50.2M 4017M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 2m ago 7m 51.6M 4017M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 2m ago 7m 45.7M 4017M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6m) 2m ago 6m 52.3M 4017M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:21:14.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (6m) 2m ago 6m 50.8M 4272M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (5m) 2m ago 5m 49.4M 4272M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (5m) 2m ago 5m 48.7M 4272M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (5m) 2m ago 5m 49.1M 4272M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (3m) 2m ago 4m 46.3M - 514e6a882f6e 8bde7995dc5e 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (2m) 2m ago 2m 69.9M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (2m) 2m ago 2m 69.2M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (2m) 2m ago 2m 72.1M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:21:14.222 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (2m) 2m ago 2m 73.0M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:21:14.584 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:21:14.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:14.586 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:21:14.586 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-06T02:21:14.586 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:21:14.586 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/23 daemons upgraded", 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c image on host smithi104" 2024-04-06T02:21:14.890 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:21:14.890 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[29567]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[29567]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[29567]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2334726472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[24844]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[24844]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[24844]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.891 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:14 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2334726472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:14 smithi104 ceph-mon[25739]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:14 smithi104 ceph-mon[25739]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:14 smithi104 ceph-mon[25739]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:14.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:14 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2334726472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:15.252 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:21:15.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:15 smithi104 ceph-mon[25739]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:15 smithi104 ceph-mon[25739]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:15 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1849810042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:15.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[24844]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[24844]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1849810042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:15.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[29567]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[29567]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:15.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:15 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1849810042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:16.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:16 smithi104 ceph-mon[25739]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:16.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:16 smithi017 ceph-mon[24844]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:16.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:16 smithi017 ceph-mon[29567]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:16.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:16] "GET /metrics HTTP/1.1" 200 214831 "" "Prometheus/2.33.4" 2024-04-06T02:21:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:17 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:17.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:17 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:17 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:18.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:18 smithi017 ceph-mon[29567]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:18 smithi017 ceph-mon[24844]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:19.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:18 smithi104 ceph-mon[25739]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:20.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:21:20.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:20 smithi104 ceph-mon[25739]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:20.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:20 smithi017 ceph-mon[29567]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:20.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:20 smithi017 ceph-mon[24844]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:22.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:22 smithi104 ceph-mon[25739]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:22 smithi017 ceph-mon[24844]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:22 smithi017 ceph-mon[29567]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:21:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:21:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:24.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:24 smithi104 ceph-mon[25739]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:24.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:24 smithi017 ceph-mon[29567]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:24 smithi017 ceph-mon[24844]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:25.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:25.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:25 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:25.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:25.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:25 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:25 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:26.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:26 smithi104 ceph-mon[25739]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:26.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:26 smithi017 ceph-mon[29567]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:26 smithi017 ceph-mon[24844]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:26.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:26] "GET /metrics HTTP/1.1" 200 214830 "" "Prometheus/2.33.4" 2024-04-06T02:21:27.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:27 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:27.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:27 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:28.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:27 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:28 smithi017 ceph-mon[24844]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:28.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:28 smithi017 ceph-mon[29567]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:29.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:28 smithi104 ceph-mon[25739]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:30.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[26918]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:21:30.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:30 smithi104 ceph-mon[25739]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:30.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:30 smithi017 ceph-mon[29567]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:30 smithi017 ceph-mon[24844]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:32.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:32 smithi104 ceph-mon[25739]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:32.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:32 smithi017 ceph-mon[29567]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:32 smithi017 ceph-mon[24844]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:33.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:21:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:33.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:33.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:21:34.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:34 smithi104 ceph-mon[25739]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:34.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:34 smithi017 ceph-mon[29567]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:34 smithi017 ceph-mon[24844]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:36.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:36 smithi104 ceph-mon[25739]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:36.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:36 smithi017 ceph-mon[29567]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:36 smithi017 ceph-mon[24844]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:36.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:36] "GET /metrics HTTP/1.1" 200 214830 "" "Prometheus/2.33.4" 2024-04-06T02:21:37.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:37 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:37.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:37 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:38.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:37 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:38.948 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:38 smithi104 ceph-mon[25739]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:38.948 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:38.948 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:21:38.948 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:21:38.948 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:38 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:38.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[29567]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:38.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:38.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[24844]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:21:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:38 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:39.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:39 smithi017 ceph-mon[29567]: Upgrade: Updating mgr.x 2024-04-06T02:21:39.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:39 smithi017 ceph-mon[29567]: Deploying daemon mgr.x on smithi104 2024-04-06T02:21:39.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:39 smithi017 ceph-mon[24844]: Upgrade: Updating mgr.x 2024-04-06T02:21:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:39 smithi017 ceph-mon[24844]: Deploying daemon mgr.x on smithi104 2024-04-06T02:21:40.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:39 smithi104 ceph-mon[25739]: Upgrade: Updating mgr.x 2024-04-06T02:21:40.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:39 smithi104 ceph-mon[25739]: Deploying daemon mgr.x on smithi104 2024-04-06T02:21:40.076 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:39 smithi104 systemd[1]: Stopping Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:21:40.076 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:39 smithi104 bash[46756]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-06T02:21:40.380 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 podman[46769]: 2024-04-06 02:21:40.092003867 +0000 UTC m=+0.153215649 container died 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, vcs-type=git, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, name=centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T02:21:40.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 podman[46769]: 2024-04-06 02:21:40.538244439 +0000 UTC m=+0.599456272 container cleanup 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, vendor=Red Hat, Inc., GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.buildah.version=1.19.8, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git, architecture=x86_64, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.component=centos-stream-container) 2024-04-06T02:21:40.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 bash[46769]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x 2024-04-06T02:21:40.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 bash[46796]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr.x" found: no such container 2024-04-06T02:21:40.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:40 smithi017 ceph-mon[29567]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:40.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:40 smithi017 ceph-mon[24844]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:41.033 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:40 smithi104 ceph-mon[25739]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:41.034 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 podman[46784]: 2024-04-06 02:21:40.839793148 +0000 UTC m=+0.744649550 container remove 247dc63ae7b6811f6062d1682e3dd1a5d04623e31347132a0b7c497082046a94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, release=754, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8) 2024-04-06T02:21:41.034 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:21:41.034 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Failed with result 'exit-code'. 2024-04-06T02:21:41.034 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 systemd[1]: Stopped Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:21:41.034 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:40 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Consumed 16.395s CPU time. 2024-04-06T02:21:41.291 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 systemd[1]: Starting Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:21:41.291 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 podman[46902]: 2024-04-06 02:21:41.167538836 +0000 UTC m=+0.018146256 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:21:41.585 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 podman[46902]: 2024-04-06 02:21:41.307337537 +0000 UTC m=+0.157944938 container create 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-06T02:21:41.585 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 podman[46902]: 2024-04-06 02:21:41.518025299 +0000 UTC m=+0.368632701 container init 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-06T02:21:41.585 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 podman[46902]: 2024-04-06 02:21:41.520644831 +0000 UTC m=+0.371252234 container start 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-06T02:21:41.585 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 ceph-mgr[46920]: -- 172.21.15.104:0/3955411196 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c75248c5a0 con 0x55c752461000 2024-04-06T02:21:41.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 bash[46902]: 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 2024-04-06T02:21:41.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:41 smithi104 systemd[1]: Started Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:21:42.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:42.119+0000 7f94bc868200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:21:42.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:42.213+0000 7f94bc868200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:21:42.880 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-mon[25739]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:42.880 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:42.880 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:42.881 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:42.881 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:21:42.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[29567]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[24844]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:21:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:42 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:21:43.143 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:42.895+0000 7f94bc868200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:21:43.143 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:42.988+0000 7f94bc868200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:21:43.143 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.129+0000 7f94bc868200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:21:43.393 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.230+0000 7f94bc868200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:21:43.393 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.321+0000 7f94bc868200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:21:43.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.588+0000 7f94bc868200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:21:43.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:21:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:43.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:21:43.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:43.991 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.726+0000 7f94bc868200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:21:43.991 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.834+0000 7f94bc868200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:21:43.991 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:43.948+0000 7f94bc868200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:21:44.301 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:44.134+0000 7f94bc868200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:21:44.901 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:44 smithi104 ceph-mon[25739]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:44.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:44 smithi017 ceph-mon[29567]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:44 smithi017 ceph-mon[24844]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:45.584 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:21:45.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:45.221+0000 7f94bc868200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:21:45.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:45.353+0000 7f94bc868200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:21:45.984 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:45.701+0000 7f94bc868200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:21:45.984 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:45.797+0000 7f94bc868200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:21:45.984 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:45.999+0000 7f94bc868200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 3m ago 4m 17.9M - ba2b418f427c e252f7611c30 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 running (3m) 3m ago 3m 49.0M - 8.3.5 dad864ee21e9 40df0b2e5224 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (3m) 3m ago 3m 74.5M - 3.5 e1d6a67b021e fba4f4b5b4eb 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283 starting - - - - 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (10m) 3m ago 10m 457M - 17.2.0 e1d6a67b021e a6a010afe065 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (10m) 3m ago 10m 48.8M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:21:46.184 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (9m) 3m ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (9m) 3m ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 3m ago 4m 18.3M - 1dbe0e931976 db0c895136c8 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (4m) 3m ago 4m 16.4M - 1dbe0e931976 ad624bfcf398 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (8m) 3m ago 8m 50.2M 4017M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (8m) 3m ago 8m 51.6M 4017M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 3m ago 7m 45.7M 4017M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (7m) 3m ago 7m 52.3M 4017M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (6m) 3m ago 6m 50.8M 4272M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (6m) 3m ago 6m 49.4M 4272M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (6m) 3m ago 6m 48.7M 4272M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (5m) 3m ago 5m 49.1M 4272M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (3m) 3m ago 5m 46.3M - 514e6a882f6e 8bde7995dc5e 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (3m) 3m ago 3m 69.9M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (3m) 3m ago 3m 69.2M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:21:46.185 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (3m) 3m ago 3m 72.1M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:21:46.186 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (3m) 3m ago 3m 73.0M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:21:46.246 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:46.196+0000 7f94bc868200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:21:46.548 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:21:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:21:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:21:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-06T02:21:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:21:46.550 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-mon[25739]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-mon[25739]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-mon[25739]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2725537042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:46.515+0000 7f94bc868200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:21:46.777 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:46.792+0000 7f94bc868200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mgr daemons" 2024-04-06T02:21:46.856 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:21:46.856 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:46] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[24844]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[24844]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[24844]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2725537042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[29567]: from='client.24991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[29567]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[29567]: from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:46.857 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:46.858 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:46 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2725537042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:21:47.215 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:21:47.350 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:47.048+0000 7f94bc868200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:47.902+0000 7f94bc868200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-mon[25739]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-mon[25739]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:48.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:47 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3664192611' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[29567]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[29567]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3664192611' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[24844]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[24844]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:47 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3664192611' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:21:48.565 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:48.484+0000 7f94bc868200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:21:48.843 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:48.584+0000 7f94bc868200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:21:48.843 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:48.812+0000 7f94bc868200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:21:48.843 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:48] ENGINE Bus STARTING 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: Standby manager daemon x restarted 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: CherryPy Checker: 2024-04-06T02:21:49.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: The Application mounted at '' has an empty config. 2024-04-06T02:21:49.163 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:21:49.163 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:48] ENGINE Serving on http://:::9283 2024-04-06T02:21:49.163 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:48] ENGINE Bus STARTED 2024-04-06T02:21:49.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:49.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: Standby manager daemon x restarted 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: Standby manager daemon x restarted 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: Standby manager daemon x started 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:21:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:48 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:21:50.133 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:49 smithi104 ceph-mon[25739]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-06T02:21:50.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:49 smithi017 ceph-mon[29567]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-06T02:21:50.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:49 smithi017 ceph-mon[24844]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-06T02:21:50.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:21:50.848 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:50 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:50.554Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:21:51.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:50 smithi104 ceph-mon[25739]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:51.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:50 smithi017 ceph-mon[29567]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:50 smithi017 ceph-mon[24844]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:53.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:52 smithi104 ceph-mon[25739]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:52 smithi017 ceph-mon[24844]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:53.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:52 smithi017 ceph-mon[29567]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:21:53.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:21:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:53.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:21:53.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:21:55.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:54 smithi104 ceph-mon[25739]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:55.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:54 smithi017 ceph-mon[29567]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:55.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:54 smithi017 ceph-mon[24844]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:56.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:21:56] "GET /metrics HTTP/1.1" 200 214838 "" "Prometheus/2.33.4" 2024-04-06T02:21:56.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[24844]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[29567]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:56 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-06T02:21:57.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:56 smithi104 ceph-mon[25739]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:21:57.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:57.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' 2024-04-06T02:21:57.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:57.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:56 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:57] ENGINE Bus STOPPING 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:57] ENGINE Bus STOPPED 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:57] ENGINE Bus STARTING 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:58] ENGINE Serving on http://:::9283 2024-04-06T02:21:58.162 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:21:58] ENGINE Bus STARTED 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: Failing over to other MGR 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: osdmap e96: 8 total, 8 up, 8 in 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: Standby manager daemon y started 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: mgrmap e22: x(active, starting, since 1.00106s), standbys: y 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:21:58.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:21:58.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:57 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:21:58.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:58.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: Failing over to other MGR 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: osdmap e96: 8 total, 8 up, 8 in 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: Standby manager daemon y started 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: mgrmap e22: x(active, starting, since 1.00106s), standbys: y 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:21:58.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:57.866+0000 7fbc47167700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setuser ceph since I am not root 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ignoring --setgroup ceph since I am not root 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mgr[25081]: -- 172.21.15.17:0/3273817595 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc32356340 con 0x55fc3232f000 2024-04-06T02:21:58.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: Failing over to other MGR 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: osdmap e96: 8 total, 8 up, 8 in 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: Standby manager daemon y started 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.14430 172.21.15.17:0/3796891466' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: mgrmap e22: x(active, starting, since 1.00106s), standbys: y 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:21:58.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:21:58.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:57 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:21:58.488 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:21:58.319Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:21:58.489 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:58.428+0000 7f8057fc5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:21:58.864 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:58.527+0000 7f8057fc5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:21:59.145 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:21:59.145 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:21:59.145 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:21:59.145 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: Manager daemon x is now available 2024-04-06T02:21:59.145 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: Queued rgw.foo for migration 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: Queued rgw.smpl for migration 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.146 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: Deploying cephadm binary to smithi104 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[29567]: Deploying cephadm binary to smithi017 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: Manager daemon x is now available 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: Queued rgw.foo for migration 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: Queued rgw.smpl for migration 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-06T02:21:59.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: Deploying cephadm binary to smithi104 2024-04-06T02:21:59.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:58 smithi017 ceph-mon[24844]: Deploying cephadm binary to smithi017 2024-04-06T02:21:59.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: Manager daemon x is now available 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: Queued rgw.foo for migration 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: Queued rgw.smpl for migration 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:21:59.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-06T02:21:59.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: Deploying cephadm binary to smithi104 2024-04-06T02:21:59.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:58 smithi104 ceph-mon[25739]: Deploying cephadm binary to smithi017 2024-04-06T02:21:59.477 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.160+0000 7f8057fc5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:21:59.477 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.252+0000 7f8057fc5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:21:59.477 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.339+0000 7f8057fc5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:21:59.477 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.491+0000 7f8057fc5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:21:59.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.574+0000 7f8057fc5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:21:59.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.689+0000 7f8057fc5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:22:00.015 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[29567]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-06T02:22:00.015 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[29567]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:00.015 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.015 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[24844]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[24844]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.775+0000 7f8057fc5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:21:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:21:59.941+0000 7f8057fc5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:22:00.016 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:00.030+0000 7f8057fc5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:22:00.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:59 smithi104 ceph-mon[25739]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-06T02:22:00.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:59 smithi104 ceph-mon[25739]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:00.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:59 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:21:59 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:00.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:00] "GET /metrics HTTP/1.1" 200 35350 "" "Prometheus/2.33.4" 2024-04-06T02:22:01.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-mon[25739]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:01.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-mon[25739]: [06/Apr/2024:02:22:00] ENGINE Bus STARTING 2024-04-06T02:22:01.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-mon[25739]: [06/Apr/2024:02:22:00] ENGINE Serving on http://172.21.15.104:8765 2024-04-06T02:22:01.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-mon[25739]: [06/Apr/2024:02:22:00] ENGINE Serving on https://172.21.15.104:7150 2024-04-06T02:22:01.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:00 smithi104 ceph-mon[25739]: [06/Apr/2024:02:22:00] ENGINE Bus STARTED 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[29567]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[29567]: [06/Apr/2024:02:22:00] ENGINE Bus STARTING 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[29567]: [06/Apr/2024:02:22:00] ENGINE Serving on http://172.21.15.104:8765 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[29567]: [06/Apr/2024:02:22:00] ENGINE Serving on https://172.21.15.104:7150 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[29567]: [06/Apr/2024:02:22:00] ENGINE Bus STARTED 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[24844]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[24844]: [06/Apr/2024:02:22:00] ENGINE Bus STARTING 2024-04-06T02:22:01.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[24844]: [06/Apr/2024:02:22:00] ENGINE Serving on http://172.21.15.104:8765 2024-04-06T02:22:01.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[24844]: [06/Apr/2024:02:22:00] ENGINE Serving on https://172.21.15.104:7150 2024-04-06T02:22:01.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:00 smithi017 ceph-mon[24844]: [06/Apr/2024:02:22:00] ENGINE Bus STARTED 2024-04-06T02:22:01.211 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.090+0000 7f8057fc5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:22:01.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.225+0000 7f8057fc5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:22:01.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.426+0000 7f8057fc5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:22:01.887 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.516+0000 7f8057fc5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:22:01.887 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.657+0000 7f8057fc5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:22:02.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:01 smithi104 ceph-mon[25739]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-06T02:22:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-mon[29567]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-06T02:22:02.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-mon[24844]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-06T02:22:02.200 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:01.985+0000 7f8057fc5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:22:02.468 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:02.215+0000 7f8057fc5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:22:02.468 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:02.304+0000 7f8057fc5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:22:02.468 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:02.483+0000 7f8057fc5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:22:03.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:02 smithi104 ceph-mon[25739]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:03.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:02 smithi017 ceph-mon[29567]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:03.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:02 smithi017 ceph-mon[24844]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:03.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:03.447+0000 7f8057fc5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:22:03.738 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:22:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:22:03.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:22:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:22:03.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:22:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:22:04.018 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:03.768+0000 7f8057fc5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:22:04.018 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:03.869+0000 7f8057fc5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:22:04.018 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:04.033+0000 7f8057fc5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:22:04] ENGINE Bus STARTING 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: CherryPy Checker: 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: The Application mounted at '' has an empty config. 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:22:04] ENGINE Serving on http://:::9283 2024-04-06T02:22:04.419 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: [06/Apr/2024:02:22:04] ENGINE Bus STARTED 2024-04-06T02:22:04.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: Standby manager daemon y restarted 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: Standby manager daemon y started 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: Standby manager daemon y restarted 2024-04-06T02:22:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: Standby manager daemon y started 2024-04-06T02:22:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:22:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:22:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:22:04.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: Standby manager daemon y restarted 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: Standby manager daemon y started 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:22:04.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:04 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/786162108' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:22:05.238 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:22:04.876Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:22:05.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:05 smithi104 ceph-mon[25739]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-06T02:22:05.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:05 smithi017 ceph-mon[29567]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-06T02:22:05.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:05 smithi017 ceph-mon[24844]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-06T02:22:06.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:06 smithi104 ceph-mon[25739]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:22:06.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:06 smithi017 ceph-mon[29567]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:22:06.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:06 smithi017 ceph-mon[24844]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:22:06.942 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:22:08.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:08 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:08.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:08.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi017 to 2737M 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:09 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi017 to 2737M 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi017 to 2737M 2024-04-06T02:22:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:09.741 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:09 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:10.353 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:10 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:10] "GET /metrics HTTP/1.1" 200 35350 "" "Prometheus/2.33.4" 2024-04-06T02:22:10.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:10 smithi104 ceph-mon[25739]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:22:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:10 smithi017 ceph-mon[29567]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:22:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:10 smithi017 ceph-mon[24844]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:22:12.661 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.104:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Adjusting osd_memory_target on smithi104 to 2992M 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:12 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Adjusting osd_memory_target on smithi104 to 2992M 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:13.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Adjusting osd_memory_target on smithi104 to 2992M 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:13.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:12 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:13.738 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:13 smithi017 systemd[1]: Stopping Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:13.739 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:13 smithi017 bash[58353]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.a" found: no such container 2024-04-06T02:22:13.739 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:13 smithi017 podman[58366]: 2024-04-06 02:22:13.527061646 +0000 UTC m=+0.125300229 container died db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:13.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=error ts=2024-04-06T02:22:13.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:22:13.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:22:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-04-06T02:22:13.739 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=warn ts=2024-04-06T02:22:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.104:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.104 because it doesn't contain any IP SANs" 2024-04-06T02:22:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: Reconfiguring daemon node-exporter.a on smithi017 2024-04-06T02:22:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:13 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:14.238 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58366]: 2024-04-06 02:22:14.015841005 +0000 UTC m=+0.614079577 container cleanup db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:14.238 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 bash[58366]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a 2024-04-06T02:22:14.238 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 bash[58394]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.a" found: no such container 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: Reconfiguring daemon node-exporter.a on smithi017 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:22:14.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: Reconfiguring daemon node-exporter.a on smithi017 2024-04-06T02:22:14.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:13 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:14.602 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58378]: 2024-04-06 02:22:14.359429731 +0000 UTC m=+0.833393091 container remove db0c895136c8ba60ca3e94835f71fd0fa5639d3f0287c0ceebc70e3db5c950c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: Stopped Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Consumed 2.685s CPU time. 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: Starting Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58461]: 2024-04-06 02:22:14.493737248 +0000 UTC m=+0.017214410 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-06T02:22:14.603 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58461]: 2024-04-06 02:22:14.619024036 +0000 UTC m=+0.142501212 container create f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58461]: 2024-04-06 02:22:14.837932392 +0000 UTC m=+0.361409528 container init f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 podman[58461]: 2024-04-06 02:22:14.840562098 +0000 UTC m=+0.364039233 container start f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.843Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.843Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.844Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.844Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.844Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.844Z caller=node_exporter.go:115 level=info collector=arp 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-06T02:22:14.866 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=edac 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-06T02:22:14.867 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-06T02:22:14.868 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=os 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=stat 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=time 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=uname 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-06T02:22:14.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-06T02:22:14.870 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-06T02:22:14.870 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-06T02:22:14.870 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a[58476]: ts=2024-04-06T02:22:14.845Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-06T02:22:15.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:14 smithi104 ceph-mon[25739]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-mon[29567]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:15.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:14 smithi017 ceph-mon[24844]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:15.238 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 bash[58461]: f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 2024-04-06T02:22:15.238 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:22:14 smithi017 systemd[1]: Started Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:16.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[24844]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:22:16.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[24844]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:22:16.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[29567]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:22:16.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:15 smithi017 ceph-mon[29567]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:22:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:15 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:15 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:16.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:15 smithi104 ceph-mon[25739]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T02:22:16.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:15 smithi104 ceph-mon[25739]: Reconfiguring daemon alertmanager.a on smithi017 2024-04-06T02:22:16.963 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:22:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:16 smithi017 ceph-mon[29567]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:16 smithi017 ceph-mon[24844]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:17.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:16 smithi104 ceph-mon[25739]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-06T02:22:17.596 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:22:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:17 smithi017 ceph-mon[24844]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:18.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:17 smithi017 ceph-mon[29567]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:18.298 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 11s ago 4m 26.9M - 0.23.0 ba2b418f427c e252f7611c30 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 running (4m) 10s ago 4m 54.9M - 8.3.5 dad864ee21e9 40df0b2e5224 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (3m) 11s ago 3m 51.4M - 3.5 e1d6a67b021e fba4f4b5b4eb 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283 running (37s) 10s ago 9m 493M - 19.0.0-2027-g9f192350 a5511c7deba9 3a23271d7c9a 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (10m) 11s ago 10m 409M - 17.2.0 e1d6a67b021e a6a010afe065 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (10m) 11s ago 10m 55.7M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (9m) 10s ago 9m 42.6M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (9m) 11s ago 9m 40.5M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 starting - - - - 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (5m) 10s ago 5m 19.4M - 1.3.1 1dbe0e931976 ad624bfcf398 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (8m) 11s ago 8m 52.8M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (8m) 11s ago 8m 55.0M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (8m) 11s ago 8m 48.0M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:22:18.299 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (7m) 11s ago 7m 55.2M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (7m) 10s ago 7m 54.1M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (7m) 10s ago 7m 52.8M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (6m) 10s ago 6m 50.7M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (6m) 10s ago 6m 52.4M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (4m) 10s ago 5m 66.0M - 2.33.4 514e6a882f6e 8bde7995dc5e 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (4m) 11s ago 3m 71.4M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (3m) 10s ago 3m 71.1M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (3m) 11s ago 3m 74.0M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:22:18.300 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (3m) 10s ago 3m 75.2M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:22:18.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:17 smithi104 ceph-mon[25739]: from='client.34655 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:22:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:22:18.718 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T02:22:19.056 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-04-06T02:22:19.057 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:22:19.057 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:22:19.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:18 smithi104 ceph-mon[25739]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:18 smithi104 ceph-mon[25739]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:19.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:18 smithi104 ceph-mon[25739]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:18 smithi104 ceph-mon[25739]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:18 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1523718516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:19.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[29567]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[29567]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:19.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[29567]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[29567]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1523718516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[24844]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[24844]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[24844]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[24844]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:18 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1523718516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:19.467 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:22:20.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:19 smithi017 ceph-mon[29567]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:20.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:19 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1273061305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:20.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:19 smithi017 ceph-mon[24844]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:19 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1273061305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:20.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:20] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-04-06T02:22:20.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:19 smithi104 ceph-mon[25739]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:20.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:19 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1273061305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:21.090 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:20 smithi017 ceph-mon[29567]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:21.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:20 smithi017 ceph-mon[24844]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:21.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:20 smithi104 ceph-mon[25739]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:22.488 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 systemd[1]: Stopping Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:22.488 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 bash[58874]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-06T02:22:22.489 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[49457]: level=info ts=2024-04-06T02:22:22.209Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:22:22.489 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 podman[58887]: 2024-04-06 02:22:22.229598694 +0000 UTC m=+0.135940503 container died e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:22.661 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.104:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:22:22.973 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 podman[58887]: 2024-04-06 02:22:22.676039898 +0000 UTC m=+0.582381746 container cleanup e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:22.973 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 bash[58887]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T02:22:22.973 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:22 smithi017 bash[58913]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-06T02:22:23.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:22 smithi017 ceph-mon[29567]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:23.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:22 smithi017 ceph-mon[24844]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:23.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58901]: 2024-04-06 02:22:23.08588126 +0000 UTC m=+0.856159657 container remove e252f7611c307462a9b60f50e781a2437cf54519c5b6542eeeccb63a82bc1d4c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:23.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58901]: 2024-04-06 02:22:23.169927928 +0000 UTC m=+0.940206323 volume remove 0b5bb52e70527c9ac0475790951d274e91e24817fb4edb9836ca97624d25f63d 2024-04-06T02:22:23.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-06T02:22:23.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 systemd[1]: Stopped Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:23.239 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 systemd[1]: Starting Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:23.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:22 smithi104 ceph-mon[25739]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:23.681 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58983]: 2024-04-06 02:22:23.308104478 +0000 UTC m=+0.018709219 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-06T02:22:23.681 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58983]: 2024-04-06 02:22:23.429264664 +0000 UTC m=+0.139869409 volume create e448428390426f70e129cee2b74c89b9d3b8c571108d28d0a40dc9fa9eb00e2c 2024-04-06T02:22:23.682 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58983]: 2024-04-06 02:22:23.529662483 +0000 UTC m=+0.240267264 container create cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58983]: 2024-04-06 02:22:23.799371281 +0000 UTC m=+0.509976019 container init cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 podman[58983]: 2024-04-06 02:22:23.802394162 +0000 UTC m=+0.512998900 container start cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.813Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-06T02:22:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.813Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-06T02:22:23.988 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.814Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.816Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.841Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.842Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.844Z caller=main.go:518 msg=Listening address=:9093 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:23.844Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 bash[58983]: cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb 2024-04-06T02:22:23.989 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:23 smithi017 systemd[1]: Started Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:24.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:24.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:24.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:24.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[29567]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:24.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:24 smithi017 ceph-mon[24844]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:25.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:24 smithi104 ceph-mon[25739]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:22:25.885 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:25.816Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000427691s 2024-04-06T02:22:26.790 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:22:27.108 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:26 smithi017 ceph-mon[29567]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:27.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:26 smithi017 ceph-mon[24844]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:26 smithi104 ceph-mon[25739]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:27.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:22:27.631+0000 7f94424cf700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:27.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Creating ceph-iscsi config... 2024-04-06T02:22:27.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:27.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return _run_code(code, main_globals, None, 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: exec(code, run_globals) 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:27.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:22:27.914 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return future.result(timeout) 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return self.__get_result() 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: raise self._exception 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: config_blobs=daemon_spec.final_config, 2024-04-06T02:22:27.915 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Creating ceph-iscsi config... 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return _run_code(code, main_globals, None, 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: exec(code, run_globals) 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:27.916 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:27.917 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[24844]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:22:29.240 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:22:29.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[29567]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:22:29.243 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:22:29.244 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.245 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.246 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[24844]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[24844]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[24844]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:29.247 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[29567]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[29567]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[29567]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:29.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:28 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:28 smithi104 ceph-mon[25739]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.412 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.413 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: return future.result(timeout) 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.__get_result() 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise self._exception 2024-04-06T02:22:29.414 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.415 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:22:29.416 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:28 smithi104 ceph-mon[25739]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:28 smithi104 ceph-mon[25739]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:28 smithi104 ceph-mon[25739]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:22:29.417 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:28 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:30.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:30] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.33.4" 2024-04-06T02:22:30.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:30 smithi104 ceph-mon[25739]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:22:30.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:30 smithi017 ceph-mon[29567]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:22:30.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:30 smithi017 ceph-mon[24844]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:22:32.661 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.104:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:22:33.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:32 smithi017 ceph-mon[29567]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:33.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:32 smithi017 ceph-mon[24844]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:33.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:32 smithi104 ceph-mon[25739]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:34.238 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:22:33 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:22:33.820Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003878359s 2024-04-06T02:22:35.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:34 smithi017 ceph-mon[29567]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:35.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:34 smithi017 ceph-mon[24844]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:35.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:34 smithi104 ceph-mon[25739]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:36.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:36 smithi017 ceph-mon[29567]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:36.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:36 smithi017 ceph-mon[24844]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:36.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[25077]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-06T02:22:37.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:36 smithi104 ceph-mon[25739]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T02:22:38.992 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:38 smithi017 ceph-mon[29567]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:38.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:38 smithi017 ceph-mon[24844]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:39.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:38 smithi104 ceph-mon[25739]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:39.473 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 systemd[1]: Stopping Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:39.473 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 bash[50411]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:22:39.909 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.489Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.490Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.490Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[42623]: ts=2024-04-06T02:22:39.490Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-06T02:22:39.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:39 smithi104 podman[50424]: 2024-04-06 02:22:39.499493668 +0000 UTC m=+0.109756156 container died 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:40.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3896184539' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:22:40.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3700577025' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:39 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.162 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50424]: 2024-04-06 02:22:40.024580145 +0000 UTC m=+0.634842632 container cleanup 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:40.162 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 bash[50424]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T02:22:40.162 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 bash[50450]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:22:40.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3896184539' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:22:40.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3700577025' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3896184539' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:22:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3700577025' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:39 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]: dispatch 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50438]: 2024-04-06 02:22:40.250920718 +0000 UTC m=+0.754000669 container remove 8bde7995dc5e21c5a9b5163b6e94a9784575b501cc545175a7cb24e01dcf08d8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 systemd[1]: Stopped Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service: Consumed 1.466s CPU time. 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 systemd[1]: Starting Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50516]: 2024-04-06 02:22:40.379658749 +0000 UTC m=+0.018728304 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-06T02:22:40.502 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50516]: 2024-04-06 02:22:40.518575185 +0000 UTC m=+0.157644726 container create 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:40.763 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50516]: 2024-04-06 02:22:40.754551314 +0000 UTC m=+0.393620855 container init 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:40.763 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 podman[50516]: 2024-04-06 02:22:40.757103693 +0000 UTC m=+0.396173243 container start 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:22:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-mon[25739]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]': finished 2024-04-06T02:22:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-mon[25739]: osdmap e97: 8 total, 8 up, 8 in 2024-04-06T02:22:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.162 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-06T02:22:41.162 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi104 (none))" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.779Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.781Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.781Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.783Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.785Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.785Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.581µs 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.785Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 bash[50516]: 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 systemd[1]: Started Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:41.163 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:40.982Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-06T02:22:41.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[29567]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:41.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]': finished 2024-04-06T02:22:41.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[29567]: osdmap e97: 8 total, 8 up, 8 in 2024-04-06T02:22:41.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[24844]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:22:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3771182315"}]': finished 2024-04-06T02:22:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[24844]: osdmap e97: 8 total, 8 up, 8 in 2024-04-06T02:22:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:40 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:41.518 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 systemd[1]: Stopping Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:41.518 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 bash[50641]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.b" found: no such container 2024-04-06T02:22:41.518 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.226Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-06T02:22:41.518 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.226Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-06T02:22:41.518 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.226Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.993µs wal_replay_duration=441.382802ms total_replay_duration=441.423569ms 2024-04-06T02:22:41.519 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.229Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T02:22:41.519 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.229Z caller=main.go:947 level=info msg="TSDB started" 2024-04-06T02:22:41.519 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.229Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T02:22:41.519 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.249Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.148886ms db_storage=792ns remote_storage=1.275µs web_handler=367ns query_engine=738ns scrape=366.776µs scrape_sd=92.848µs notify=14.302µs notify_sd=14.774µs rules=19.326373ms 2024-04-06T02:22:41.519 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:41.249Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-06T02:22:41.782 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 podman[50654]: 2024-04-06 02:22:41.534636307 +0000 UTC m=+0.110102598 container died ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:42.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-mon[25739]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-06T02:22:42.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-mon[25739]: Reconfiguring daemon node-exporter.b on smithi104 2024-04-06T02:22:42.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1852261385' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:41 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 podman[50654]: 2024-04-06 02:22:41.974662807 +0000 UTC m=+0.550129095 container cleanup ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:42.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 bash[50654]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b 2024-04-06T02:22:42.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:41 smithi104 bash[50683]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.b" found: no such container 2024-04-06T02:22:42.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[29567]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-06T02:22:42.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[29567]: Reconfiguring daemon node-exporter.b on smithi104 2024-04-06T02:22:42.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1852261385' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[24844]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-06T02:22:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[24844]: Reconfiguring daemon node-exporter.b on smithi104 2024-04-06T02:22:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1852261385' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:41 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]: dispatch 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 podman[50668]: 2024-04-06 02:22:42.317960348 +0000 UTC m=+0.787953140 container remove ad624bfcf3980bd33dcd0558f2377c1f667179dd155d4b5b62b03cab2faebee8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: Stopped Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Consumed 2.731s CPU time. 2024-04-06T02:22:42.541 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: Starting Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:42.794 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 podman[50762]: 2024-04-06 02:22:42.461519028 +0000 UTC m=+0.021491267 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-06T02:22:42.795 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 podman[50762]: 2024-04-06 02:22:42.593751606 +0000 UTC m=+0.153723834 container create e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:42.795 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 podman[50762]: 2024-04-06 02:22:42.80426041 +0000 UTC m=+0.364232636 container init e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:42.795 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 podman[50762]: 2024-04-06 02:22:42.807562917 +0000 UTC m=+0.367535145 container start e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:22:43.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:43.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]': finished 2024-04-06T02:22:43.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: osdmap e98: 8 total, 8 up, 8 in 2024-04-06T02:22:43.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]: dispatch 2024-04-06T02:22:43.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:43.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.810Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.811Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.811Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.811Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=arp 2024-04-06T02:22:43.162 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=edac 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-06T02:22:43.163 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-06T02:22:43.164 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=os 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=stat 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=time 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-06T02:22:43.165 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=uname 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b[50777]: ts=2024-04-06T02:22:42.812Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 bash[50762]: e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a 2024-04-06T02:22:43.166 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:22:42 smithi104 systemd[1]: Started Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:43.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:43.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]': finished 2024-04-06T02:22:43.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: osdmap e98: 8 total, 8 up, 8 in 2024-04-06T02:22:43.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]: dispatch 2024-04-06T02:22:43.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2724707203"}]': finished 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: osdmap e98: 8 total, 8 up, 8 in 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]: dispatch 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:42 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]': finished 2024-04-06T02:22:44.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: osdmap e99: 8 total, 8 up, 8 in 2024-04-06T02:22:44.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: Reconfiguring grafana.a (dependencies changed)... 2024-04-06T02:22:44.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T02:22:44.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: Reconfiguring daemon grafana.a on smithi104 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]: dispatch 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]': finished 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: osdmap e99: 8 total, 8 up, 8 in 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: Reconfiguring grafana.a (dependencies changed)... 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T02:22:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: Reconfiguring daemon grafana.a on smithi104 2024-04-06T02:22:44.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:43 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]: dispatch 2024-04-06T02:22:44.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3388152935' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/611295603"}]': finished 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: osdmap e99: 8 total, 8 up, 8 in 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: Reconfiguring grafana.a (dependencies changed)... 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: Reconfiguring daemon grafana.a on smithi104 2024-04-06T02:22:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:43 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]: dispatch 2024-04-06T02:22:45.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:45 smithi104 ceph-mon[25739]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:45.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:45 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]': finished 2024-04-06T02:22:45.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:45 smithi104 ceph-mon[25739]: osdmap e100: 8 total, 8 up, 8 in 2024-04-06T02:22:45.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:45 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3579047101' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:45.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:45 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[29567]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]': finished 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[29567]: osdmap e100: 8 total, 8 up, 8 in 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3579047101' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[24844]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:22:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1735733947' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/611295603"}]': finished 2024-04-06T02:22:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[24844]: osdmap e100: 8 total, 8 up, 8 in 2024-04-06T02:22:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3579047101' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:45 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]: dispatch 2024-04-06T02:22:46.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:46 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]': finished 2024-04-06T02:22:46.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:46 smithi104 ceph-mon[25739]: osdmap e101: 8 total, 8 up, 8 in 2024-04-06T02:22:46.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:46 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]: dispatch 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]': finished 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[29567]: osdmap e101: 8 total, 8 up, 8 in 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]: dispatch 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/476672348"}]': finished 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[24844]: osdmap e101: 8 total, 8 up, 8 in 2024-04-06T02:22:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:46 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]: dispatch 2024-04-06T02:22:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:47 smithi104 ceph-mon[25739]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:47 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]': finished 2024-04-06T02:22:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:47 smithi104 ceph-mon[25739]: osdmap e102: 8 total, 8 up, 8 in 2024-04-06T02:22:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:47 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2935017034' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:47 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:47.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[29567]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:47.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]': finished 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[29567]: osdmap e102: 8 total, 8 up, 8 in 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2935017034' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[24844]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/4177844234' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1233898327"}]': finished 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[24844]: osdmap e102: 8 total, 8 up, 8 in 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2935017034' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:47 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]: dispatch 2024-04-06T02:22:48.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:48 smithi104 ceph-mon[25739]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]': finished 2024-04-06T02:22:48.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:48 smithi104 ceph-mon[25739]: osdmap e103: 8 total, 8 up, 8 in 2024-04-06T02:22:48.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:48 smithi017 ceph-mon[29567]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]': finished 2024-04-06T02:22:48.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:48 smithi017 ceph-mon[29567]: osdmap e103: 8 total, 8 up, 8 in 2024-04-06T02:22:48.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:48 smithi017 ceph-mon[24844]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/136955192"}]': finished 2024-04-06T02:22:48.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:48 smithi017 ceph-mon[24844]: osdmap e103: 8 total, 8 up, 8 in 2024-04-06T02:22:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:49 smithi104 ceph-mon[25739]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:49.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:49 smithi017 ceph-mon[29567]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:49.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:49 smithi017 ceph-mon[24844]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:22:49.835 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:22:50.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:50 smithi017 ceph-mon[29567]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:50.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:50 smithi017 ceph-mon[24844]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:50.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:50 smithi104 ceph-mon[25739]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:22:50.505 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 starting - - - - 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 running (5m) 43s ago 5m 54.9M - 8.3.5 dad864ee21e9 40df0b2e5224 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (4m) 43s ago 4m 51.4M - 3.5 e1d6a67b021e fba4f4b5b4eb 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283 running (69s) 43s ago 9m 493M - 19.0.0-2027-g9f192350 a5511c7deba9 3a23271d7c9a 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (11m) 43s ago 11m 409M - 17.2.0 e1d6a67b021e a6a010afe065 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (11m) 43s ago 11m 55.7M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (10m) 43s ago 10m 42.6M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (10m) 43s ago 10m 40.5M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 starting - - - - 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 starting - - - - 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (9m) 43s ago 9m 52.8M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:22:50.506 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (9m) 43s ago 9m 55.0M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (8m) 43s ago 8m 48.0M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (8m) 43s ago 8m 55.2M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (7m) 43s ago 7m 54.1M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (7m) 43s ago 7m 52.8M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (7m) 43s ago 7m 50.7M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (6m) 43s ago 6m 52.4M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 starting - - - - 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (4m) 43s ago 4m 71.4M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (4m) 43s ago 4m 71.1M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (4m) 43s ago 4m 74.0M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:22:50.507 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (4m) 43s ago 4m 75.2M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:22:50.908 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:22:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:22:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:22:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:50.908 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:22:50.909 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:22:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-06T02:22:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:22:50.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:22:50.910 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:22:51.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[24844]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[24844]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-04-06T02:22:51.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[24844]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[24844]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1855615619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[29567]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[29567]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[29567]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[29567]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:51 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1855615619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-04-06T02:22:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/23 daemons upgraded", 2024-04-06T02:22:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-04-06T02:22:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:22:51.248 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:22:51.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:22:51] "GET /metrics HTTP/1.1" 200 38156 "" "Prometheus/2.33.4" 2024-04-06T02:22:51.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-mon[25739]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-mon[25739]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-04-06T02:22:51.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-mon[25739]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-mon[25739]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:51.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:51 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1855615619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:22:51.656 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:22:52.146 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:52 smithi104 ceph-mon[25739]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:52.147 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:52 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3547676232' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:52 smithi017 ceph-mon[29567]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:52 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3547676232' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:52 smithi017 ceph-mon[24844]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:22:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:52 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3547676232' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:22:53.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:53 smithi104 ceph-mon[25739]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-06T02:22:53.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:53 smithi017 ceph-mon[29567]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-06T02:22:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:53 smithi017 ceph-mon[24844]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-06T02:22:54.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 systemd[1]: Stopping Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:54.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 bash[51068]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana.a" found: no such container 2024-04-06T02:22:54.570 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[41818]: t=2024-04-06T02:22:54+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-06T02:22:54.570 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 podman[51081]: 2024-04-06 02:22:54.286395732 +0000 UTC m=+0.116243294 container died 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.component=ubi8-container, name=ubi8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-type=git, distribution-scope=public, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-06T02:22:54.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 podman[51081]: 2024-04-06 02:22:54.659539978 +0000 UTC m=+0.489387542 container cleanup 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , version=8.5, name=ubi8, distribution-scope=public, vcs-type=git, release=236.1648460182, com.redhat.component=ubi8-container, architecture=x86_64) 2024-04-06T02:22:54.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 bash[51081]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a 2024-04-06T02:22:54.912 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 bash[51107]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana.a" found: no such container 2024-04-06T02:22:55.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-mon[25739]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:22:55.311 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 podman[51095]: 2024-04-06 02:22:54.960482563 +0000 UTC m=+0.674446069 container remove 40df0b2e522415f3fb9057afa29d576382124a9a5a9207a2624d162bfa3ba7c3 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, release=236.1648460182, io.openshift.tags=base rhel8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, name=ubi8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:22:55.311 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-04-06T02:22:55.311 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:55.313 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Consumed 2.536s CPU time. 2024-04-06T02:22:55.313 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:54 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:55.313 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 podman[51175]: 2024-04-06 02:22:55.203313953 +0000 UTC m=+0.114610943 container create 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.openshift.expose-services=, vcs-type=git, com.redhat.component=ubi8-container, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T02:22:55.313 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 podman[51175]: 2024-04-06 02:22:55.106322748 +0000 UTC m=+0.017619704 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:22:55.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:55 smithi017 ceph-mon[29567]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:22:55.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:55 smithi017 ceph-mon[24844]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 podman[51175]: 2024-04-06 02:22:55.447257883 +0000 UTC m=+0.358554831 container init 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 podman[51175]: 2024-04-06 02:22:55.449874711 +0000 UTC m=+0.361171667 container start 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.openshift.expose-services=, name=ubi8, version=8.5, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[51190]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[51190]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 bash[51175]: 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:22:55.600 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:55 smithi104 podman[51206]: 2024-04-06 02:22:55.555726585 +0000 UTC m=+0.072738309 container died 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, release=236.1648460182, vendor=Red Hat, Inc., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, name=ubi8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPING 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPED 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTING 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Serving on http://:::9283 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTED 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPING 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPED 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTING 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Serving on http://:::9283 2024-04-06T02:22:55.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTED 2024-04-06T02:22:55.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPING 2024-04-06T02:22:55.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:22:55.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STOPPED 2024-04-06T02:22:55.913 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTING 2024-04-06T02:22:56.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Serving on http://:::9283 2024-04-06T02:22:56.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:22:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:22:55] ENGINE Bus STARTED 2024-04-06T02:22:56.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:56 smithi104 podman[51206]: 2024-04-06 02:22:56.351416385 +0000 UTC m=+0.868428112 container remove 87d751b8e53e590dba6092c5f8908b55b2e7966c169a07ef719f3baac54a342e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, maintainer=Paul Cuzner , description=Ceph Grafana Container, vendor=Red Hat, Inc., architecture=x86_64, release=236.1648460182, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-type=git, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, distribution-scope=public) 2024-04-06T02:22:56.662 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:56 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:22:56.662 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:22:56 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:56 smithi104 ceph-mon[25739]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[29567]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:22:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:22:56.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:56 smithi017 ceph-mon[24844]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: Upgrade: Updating mgr.y 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: Deploying daemon mgr.y on smithi017 2024-04-06T02:22:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: Upgrade: Updating mgr.y 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: Deploying daemon mgr.y on smithi017 2024-04-06T02:22:58.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:58 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: Upgrade: Updating mgr.y 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: Deploying daemon mgr.y on smithi017 2024-04-06T02:22:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:22:58.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:22:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:22:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:22:59.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:22:59 smithi017 ceph-mon[24844]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-04-06T02:22:59.739 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:59 smithi017 systemd[1]: Stopping Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:22:59.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:22:59 smithi017 ceph-mon[29567]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-04-06T02:22:59.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:22:59 smithi104 ceph-mon[25739]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-04-06T02:23:00.036 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:22:59 smithi017 podman[60088]: 2024-04-06 02:22:59.789938483 +0000 UTC m=+0.156194570 container died a6a010afe065751f60d2a0a7afe7510408d13d664d579e6f39d1ec742af86dce (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, vendor=Red Hat, Inc., release=754, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-04-06T02:23:00.420 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 podman[60088]: 2024-04-06 02:23:00.201300762 +0000 UTC m=+0.567556846 container cleanup a6a010afe065751f60d2a0a7afe7510408d13d664d579e6f39d1ec742af86dce (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, name=centos-stream, release=754, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git) 2024-04-06T02:23:00.420 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 bash[60088]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:00 smithi017 ceph-mon[29567]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:00 smithi017 ceph-mon[24844]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 podman[60102]: 2024-04-06 02:23:00.436194367 +0000 UTC m=+0.651634466 container remove a6a010afe065751f60d2a0a7afe7510408d13d664d579e6f39d1ec742af86dce (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.buildah.version=1.19.8, release=754, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, version=8, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service: Failed with result 'exit-code'. 2024-04-06T02:23:00.675 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 systemd[1]: Stopped Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:00.676 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service: Consumed 55.587s CPU time. 2024-04-06T02:23:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:00 smithi104 ceph-mon[25739]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:00.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:00 smithi017 systemd[1]: Starting Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:01.345 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 podman[60234]: 2024-04-06 02:23:01.095691462 +0000 UTC m=+0.112164682 container create fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-06T02:23:01.346 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 podman[60234]: 2024-04-06 02:23:01.0018819 +0000 UTC m=+0.018355085 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:23:01.346 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 podman[60234]: 2024-04-06 02:23:01.331320668 +0000 UTC m=+0.347793843 container init fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:23:01.346 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 podman[60234]: 2024-04-06 02:23:01.334171212 +0000 UTC m=+0.350644386 container start fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-06T02:23:01.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:01 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:23:01] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.33.4" 2024-04-06T02:23:01.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 ceph-mgr[60263]: -- 172.21.15.17:0/1670409930 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3701b05a0 con 0x55c370185000 2024-04-06T02:23:01.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 bash[60234]: fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 2024-04-06T02:23:01.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 systemd[1]: Started Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:01.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:01 smithi017 ceph-mon[29567]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:01 smithi017 ceph-mon[24844]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:01.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:01 smithi104 ceph-mon[25739]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:02.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:01.919+0000 7f949a577200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:23:02.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:02.017+0000 7f949a577200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:23:02.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:02 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:02 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:02 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:02.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:02 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:02 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:02.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:02.716+0000 7f949a577200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:23:02.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:02.812+0000 7f949a577200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:23:03.200 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:02.962+0000 7f949a577200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:23:03.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.061+0000 7f949a577200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:23:03.243 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.148+0000 7f949a577200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:23:03.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.420+0000 7f949a577200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:23:03.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:03 smithi104 ceph-mon[25739]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:03.913 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-mon[29567]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:03.914 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-mon[24844]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:03.914 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.534+0000 7f949a577200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:03.914 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.631+0000 7f949a577200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:23:03.914 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.739+0000 7f949a577200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:23:03.914 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:03 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:03.929+0000 7f949a577200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:23:04.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:04.952+0000 7f949a577200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:23:05.423 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:05.087+0000 7f949a577200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:23:05.423 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:05.438+0000 7f949a577200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:23:05.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-mon[29567]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:05.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:05.532+0000 7f949a577200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:23:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-mon[24844]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:05.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:05 smithi104 ceph-mon[25739]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:05.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:05.742+0000 7f949a577200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:05.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:05 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:05.934+0000 7f949a577200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:23:06.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:06.243+0000 7f949a577200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:06.751 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:06.514+0000 7f949a577200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:06.784 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-04-06T02:23:06.784 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:06.784 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:06.784 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 podman[51925]: 2024-04-06 02:23:06.561074393 +0000 UTC m=+0.017263864 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:23:06.784 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 podman[51925]: 2024-04-06 02:23:06.674785229 +0000 UTC m=+0.130974708 container create a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, io.buildah.version=1.24.2, name=ubi8, version=8.5, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182) 2024-04-06T02:23:07.007 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:06.766+0000 7f949a577200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:23:07.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:07 smithi104 ceph-mon[25739]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:07.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:07 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:07 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:07 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:07 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 podman[51925]: 2024-04-06 02:23:06.910340393 +0000 UTC m=+0.366529823 container init a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.openshift.tags=base rhel8, distribution-scope=public, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 podman[51925]: 2024-04-06 02:23:06.91296238 +0000 UTC m=+0.369151811 container start a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[51940]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[51940]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 bash[51925]: a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f 2024-04-06T02:23:07.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:07.163 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 podman[51956]: 2024-04-06 02:23:07.021788179 +0000 UTC m=+0.072107155 container died a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, distribution-scope=public) 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[29567]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[24844]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:07.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:07 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:07.637+0000 7f949a577200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:23:08.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 podman[51956]: 2024-04-06 02:23:07.680917475 +0000 UTC m=+0.731236446 container remove a6eeea5796ab994d7be8382a275d27e298cfd267f1dd82b8979ad10e7c61b42f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, release=236.1648460182, name=ubi8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , version=8.5, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:23:08.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:23:08.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:07 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:23:08.232 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.232 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.233 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:08.247+0000 7f949a577200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:23:08.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:08.350+0000 7f949a577200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:23:08.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:08 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:08.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:08.574+0000 7f949a577200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:23:08.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:08] ENGINE Bus STARTING 2024-04-06T02:23:08.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: CherryPy Checker: 2024-04-06T02:23:08.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: The Application mounted at '' has an empty config. 2024-04-06T02:23:08.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:08.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:08] ENGINE Serving on http://:::9283 2024-04-06T02:23:08.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:08] ENGINE Bus STARTED 2024-04-06T02:23:08.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:23:09.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:09.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: Standby manager daemon y restarted 2024-04-06T02:23:09.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: Standby manager daemon y started 2024-04-06T02:23:09.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: Standby manager daemon y restarted 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: Standby manager daemon y started 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:23:09.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:09 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:09.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:09.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: Standby manager daemon y restarted 2024-04-06T02:23:09.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: Standby manager daemon y started 2024-04-06T02:23:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-06T02:23:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-06T02:23:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:09 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:10.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:10 smithi017 ceph-mon[29567]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:10.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:10 smithi017 ceph-mon[29567]: mgrmap e26: x(active, since 72s), standbys: y 2024-04-06T02:23:10.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:10 smithi017 ceph-mon[24844]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:10.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:10 smithi017 ceph-mon[24844]: mgrmap e26: x(active, since 72s), standbys: y 2024-04-06T02:23:10.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:10 smithi104 ceph-mon[25739]: from='client.34946 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:10.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:10 smithi104 ceph-mon[25739]: mgrmap e26: x(active, since 72s), standbys: y 2024-04-06T02:23:11.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:23:11] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.33.4" 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:11 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:23:11.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:11 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:12.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[29567]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:12.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[29567]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:23:12.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[29567]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:23:12.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[29567]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[24844]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[24844]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:23:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[24844]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:23:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:12 smithi017 ceph-mon[24844]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:12.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:12 smithi104 ceph-mon[25739]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:12.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:12 smithi104 ceph-mon[25739]: Reconfiguring iscsi.foo.smithi017.kcchvr (dependencies changed)... 2024-04-06T02:23:12.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:12 smithi104 ceph-mon[25739]: Reconfiguring daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:23:12.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:12 smithi104 ceph-mon[25739]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[24844]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:14.169 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:14.169 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:14.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[29567]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:14.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:14.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:13 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:14.294 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:13 smithi104 ceph-mon[25739]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-06T02:23:14.294 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:13 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' 2024-04-06T02:23:14.294 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:13 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:14.310+0000 7f94424cf700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Creating ceph-iscsi config... 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:23:14.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return _run_code(code, main_globals, None, 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: exec(code, run_globals) 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:14.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return future.result(timeout) 2024-04-06T02:23:14.664 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return self.__get_result() 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: raise self._exception 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: config_blobs=daemon_spec.final_config, 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Creating ceph-iscsi config... 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:14.665 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: Traceback (most recent call last): 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: return _run_code(code, main_globals, None, 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: exec(code, run_globals) 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:14.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.228 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.228 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.229 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:23:15.230 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: return future.result(timeout) 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.__get_result() 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise self._exception 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:23:15.231 INFO:journalctl@ceph.mon.b.smithi104.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.232 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:14.993+0000 7f94afe5c700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ignoring --setuser ceph since I am not root 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: ignoring --setgroup ceph since I am not root 2024-04-06T02:23:15.233 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:15 smithi104 ceph-mgr[46920]: -- 172.21.15.104:0/76643517 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562a92d525a0 con 0x562a92d27000 2024-04-06T02:23:15.234 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.234 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: Failing over to other MGR 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:14 smithi104 ceph-mon[25739]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:23:15.238 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.239 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.240 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.241 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.242 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-06T02:23:15.243 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:23:15.244 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:23:15.245 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE Bus STOPPING 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE Bus STOPPED 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE Bus STARTING 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE Serving on http://:::9283 2024-04-06T02:23:15.246 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:23:15] ENGINE Bus STARTED 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: Failing over to other MGR 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[24844]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1072, in deploy_daemon 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:23:15.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: Failing over to other MGR 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: from='mgr.34841 172.21.15.104:0/263236711' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-06T02:23:15.249 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:14 smithi017 ceph-mon[29567]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:15.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:15.569+0000 7f04d6bb1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:23:15.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:15.666+0000 7f04d6bb1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.34841 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: mgrmap e27: y(active, starting, since 0.6702s) 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:23:16.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: Manager daemon y is now available 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.34841 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: mgrmap e27: y(active, starting, since 0.6702s) 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:23:16.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: Manager daemon y is now available 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:16 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.34841 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: mgrmap e27: y(active, starting, since 0.6702s) 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:23:16.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: Manager daemon y is now available 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:16.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:16.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:23:16.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:23:16.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:16.344+0000 7f04d6bb1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:23:16.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:16.437+0000 7f04d6bb1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:23:16.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:16.580+0000 7f04d6bb1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:23:16.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:16.676+0000 7f04d6bb1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:23:16.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:16.784+0000 7f04d6bb1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: [06/Apr/2024:02:23:15] ENGINE Bus STARTING 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: [06/Apr/2024:02:23:15] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: [06/Apr/2024:02:23:15] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: [06/Apr/2024:02:23:15] ENGINE Bus STARTED 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: mgrmap e28: y(active, since 1.67994s) 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[29567]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: [06/Apr/2024:02:23:15] ENGINE Bus STARTING 2024-04-06T02:23:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: [06/Apr/2024:02:23:15] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:23:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: [06/Apr/2024:02:23:15] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:23:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: [06/Apr/2024:02:23:15] ENGINE Bus STARTED 2024-04-06T02:23:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: mgrmap e28: y(active, since 1.67994s) 2024-04-06T02:23:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:17 smithi017 ceph-mon[24844]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:17.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: [06/Apr/2024:02:23:15] ENGINE Bus STARTING 2024-04-06T02:23:17.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: [06/Apr/2024:02:23:15] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:23:17.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: [06/Apr/2024:02:23:15] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:23:17.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: [06/Apr/2024:02:23:15] ENGINE Bus STARTED 2024-04-06T02:23:17.253 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: mgrmap e28: y(active, since 1.67994s) 2024-04-06T02:23:17.253 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-mon[25739]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:17.253 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:17.057+0000 7f04d6bb1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:23:17.253 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:17.181+0000 7f04d6bb1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:17.253 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:17.268+0000 7f04d6bb1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:23:17.539 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:17.369+0000 7f04d6bb1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:23:17.539 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:17.554+0000 7f04d6bb1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:23:17.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:17 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-04-06T02:23:17.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:17 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:17.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:17 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[29567]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[29567]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[29567]: Cluster is now healthy 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[29567]: mgrmap e29: y(active, since 2s) 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[24844]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[24844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[24844]: Cluster is now healthy 2024-04-06T02:23:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:18 smithi017 ceph-mon[24844]: mgrmap e29: y(active, since 2s) 2024-04-06T02:23:18.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-mon[25739]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:18.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-mon[25739]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-06T02:23:18.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-mon[25739]: Cluster is now healthy 2024-04-06T02:23:18.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-mon[25739]: mgrmap e29: y(active, since 2s) 2024-04-06T02:23:18.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:17 smithi104 podman[52410]: 2024-04-06 02:23:17.877902104 +0000 UTC m=+0.024833033 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:23:18.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 podman[52410]: 2024-04-06 02:23:18.015284721 +0000 UTC m=+0.162215630 container create a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.openshift.expose-services=, vcs-type=git, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:23:18.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 podman[52410]: 2024-04-06 02:23:18.267728962 +0000 UTC m=+0.414659877 container init a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, version=8.5, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.openshift.expose-services=, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, build-date=2022-03-28T10:36:18.413762) 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 podman[52410]: 2024-04-06 02:23:18.270908219 +0000 UTC m=+0.417839123 container start a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, version=8.5, com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.24.2, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64) 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[52456]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[52456]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 bash[52410]: a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:18.524 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 podman[52486]: 2024-04-06 02:23:18.402979011 +0000 UTC m=+0.098101645 container died a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , name=ubi8, vcs-type=git, io.openshift.expose-services=, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-04-06T02:23:18.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:23:18.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:18.631+0000 7f04d6bb1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:23:18.912 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:18.766+0000 7f04d6bb1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:23:19.218 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 podman[52486]: 2024-04-06 02:23:18.97948344 +0000 UTC m=+0.674606066 container remove a60776f1ace9931fe74f2e037399d28a10d8fb322c3ba7a3e88f7dd3fdfe7208 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vcs-type=git, version=8.5, vendor=Red Hat, Inc., release=236.1648460182, io.openshift.expose-services=) 2024-04-06T02:23:19.218 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:18 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:23:19.218 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:19 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:23:19.218 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:19.116+0000 7f04d6bb1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:23:19.218 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:19.208+0000 7f04d6bb1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:23:19.589 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:19.417+0000 7f04d6bb1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:19.589 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:19.605+0000 7f04d6bb1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:23:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-mon[25739]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-mon[25739]: mgrmap e30: y(active, since 4s) 2024-04-06T02:23:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:20.160 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:19.909+0000 7f04d6bb1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:20.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:20.175+0000 7f04d6bb1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:20.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:20.415+0000 7f04d6bb1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[29567]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[29567]: mgrmap e30: y(active, since 4s) 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[24844]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[24844]: mgrmap e30: y(active, since 4s) 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:20.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:20 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.645 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:21 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:21.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:21.309+0000 7f04d6bb1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:23:22.015 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:23:22.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:21.908+0000 7f04d6bb1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:23:22.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:22.011+0000 7f04d6bb1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: Standby manager daemon x started 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:22.558 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:22 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:22.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/1816201796' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:22.227+0000 7f04d6bb1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:23:22] ENGINE Bus STARTING 2024-04-06T02:23:22.662 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: CherryPy Checker: 2024-04-06T02:23:22.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: The Application mounted at '' has an empty config. 2024-04-06T02:23:22.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: 2024-04-06T02:23:22.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:23:22] ENGINE Serving on http://:::9283 2024-04-06T02:23:22.663 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[46916]: [06/Apr/2024:02:23:22] ENGINE Bus STARTED 2024-04-06T02:23:22.692 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:23:22.692 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (59s) 3s ago 5m 13.6M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 1s ago 5m - - 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 unknown 3s ago 4m - - 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283 running (101s) 1s ago 10m 74.8M - 19.0.0-2027-g9f192350 a5511c7deba9 3a23271d7c9a 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (21s) 3s ago 11m 475M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (11m) 3s ago 12m 57.2M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (10m) 1s ago 10m 45.2M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (10m) 3s ago 10m 42.6M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (68s) 3s ago 6m 9.89M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (40s) 1s ago 6m 9587k - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (10m) 3s ago 10m 53.7M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (9m) 3s ago 9m 56.1M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (9m) 3s ago 9m 48.9M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (8m) 3s ago 8m 54.9M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (8m) 1s ago 8m 55.0M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:23:22.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (8m) 1s ago 8m 53.9M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (7m) 1s ago 7m 51.7M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (7m) 1s ago 7m 53.5M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (42s) 1s ago 6m 53.6M - 2.33.4 514e6a882f6e 97f5433fff26 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (5m) 3s ago 5m 72.1M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (5m) 1s ago 5m 71.9M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (4m) 3s ago 4m 74.8M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:23:22.694 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (4m) 1s ago 4m 76.0M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:23:23.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:23:23.105 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/23 daemons upgraded", 2024-04-06T02:23:23.475 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-04-06T02:23:23.476 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:23:23.476 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:23:23.476 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[24844]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.476 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[24844]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.476 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[24844]: mgrmap e31: y(active, since 8s), standbys: x 2024-04-06T02:23:23.476 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:23:23.476 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/1979609979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:23.477 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[29567]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.477 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[29567]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.477 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[29567]: mgrmap e31: y(active, since 8s), standbys: x 2024-04-06T02:23:23.477 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:23:23.477 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:23 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/1979609979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:23 smithi104 ceph-mon[25739]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:23 smithi104 ceph-mon[25739]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:23 smithi104 ceph-mon[25739]: mgrmap e31: y(active, since 8s), standbys: x 2024-04-06T02:23:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:23 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:23:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:23 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/1979609979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:23.881 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:23:24.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='client.25243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:24.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-06T02:23:24.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/2008592801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:23:24.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='client.25243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/2008592801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:24.562 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:24 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='client.25243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/2008592801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:24.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:24.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:24 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:25.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:25.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:23:25.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:23:25.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:25.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:25 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:25.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:23:25.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:23:25.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:23:25.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:23:25.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:25.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:25 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:26.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:26 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3269561553' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:23:26.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:26.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[24844]: from='client.? 172.21.15.17:0/3269561553' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:26.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:23:26.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:26 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3269561553' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:23:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:27 smithi104 ceph-mon[25739]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:27 smithi104 ceph-mon[25739]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:23:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:27 smithi104 ceph-mon[25739]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[24844]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[24844]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[24844]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[29567]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[29567]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T02:23:27.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:27 smithi017 ceph-mon[29567]: Reconfiguring daemon prometheus.a on smithi104 2024-04-06T02:23:28.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:28 smithi104 ceph-mon[25739]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:28.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:28 smithi017 ceph-mon[24844]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:28.640 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:28 smithi017 ceph-mon[29567]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:29.371 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 systemd[1]: Stopping Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:29.372 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 bash[53964]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:23:29.372 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:29 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-04-06T02:23:29.372 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:29 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:29.372 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:29 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:29.661 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.387Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.388Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[50531]: ts=2024-04-06T02:23:29.388Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-06T02:23:29.662 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 podman[53986]: 2024-04-06 02:23:29.397449543 +0000 UTC m=+0.146292007 container died 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:30.046 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 podman[53986]: 2024-04-06 02:23:29.761792716 +0000 UTC m=+0.510635175 container cleanup 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:30.046 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 bash[53986]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T02:23:30.047 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:29 smithi104 bash[54036]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:23:30.329 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54021]: 2024-04-06 02:23:30.063334353 +0000 UTC m=+0.668571203 container remove 97f5433fff265a1b9773a1b209e8365bed6598975c3318f1b22fe32c125a9770 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:30.329 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:30 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-06T02:23:30.329 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:30 smithi104 systemd[1]: Stopped Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:30.329 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:30 smithi104 systemd[1]: Starting Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:30.330 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54053]: 2024-04-06 02:23:30.083686623 +0000 UTC m=+0.247217118 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:23:30.330 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54053]: 2024-04-06 02:23:30.230975271 +0000 UTC m=+0.394505753 container create 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, maintainer=Paul Cuzner , io.buildah.version=1.24.2, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, release=236.1648460182, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, version=8.5, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:23:30.451 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54053]: 2024-04-06 02:23:30.466557445 +0000 UTC m=+0.630087926 container init 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, maintainer=Paul Cuzner , description=Ceph Grafana Container, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.openshift.expose-services=, io.buildah.version=1.24.2, architecture=x86_64, distribution-scope=public, io.openshift.tags=base rhel8) 2024-04-06T02:23:30.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:30 smithi104 ceph-mon[25739]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:30.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:30 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:30.829 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54053]: 2024-04-06 02:23:30.469561128 +0000 UTC m=+0.633091620 container start 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.buildah.version=1.24.2, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-type=git, version=8.5, maintainer=Paul Cuzner , release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:23:30.829 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[54084]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:30.829 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[54084]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:30.830 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 bash[54053]: 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 2024-04-06T02:23:30.830 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:30.830 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:30 smithi104 podman[54125]: 2024-04-06 02:23:30.578449769 +0000 UTC m=+0.071764494 container died 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.openshift.expose-services=, version=8.5, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container) 2024-04-06T02:23:30.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:30 smithi017 ceph-mon[29567]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:30.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:30 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:30.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:30 smithi017 ceph-mon[24844]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:30.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:30 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:31.484 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 podman[54155]: 2024-04-06 02:23:31.247149858 +0000 UTC m=+0.020197726 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-06T02:23:31.484 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 podman[54155]: 2024-04-06 02:23:31.380040681 +0000 UTC m=+0.153088537 container create 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:31.484 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:31 smithi104 podman[54125]: 2024-04-06 02:23:31.187070676 +0000 UTC m=+0.680385382 container remove 21811daf9688e145b996775234b420070b80a846d5b04976ac1aa2dc1c8793a8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, distribution-scope=public, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-type=git) 2024-04-06T02:23:31.484 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:31 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:23:31.484 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:31 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 podman[54155]: 2024-04-06 02:23:31.598930493 +0000 UTC m=+0.371978360 container init 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 podman[54155]: 2024-04-06 02:23:31.601601531 +0000 UTC m=+0.374649389 container start 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi104 (none))" 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.627Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.629Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.629Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-06T02:23:31.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.630Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.633Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.633Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.848µs 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.633Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 bash[54155]: 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 systemd[1]: Started Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:31.913 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:31.839Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-06T02:23:32.370 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.096Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-06T02:23:32.370 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.134Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-06T02:23:32.370 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.134Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-06T02:23:32.370 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.134Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=53.545µs wal_replay_duration=501.386958ms total_replay_duration=501.462258ms 2024-04-06T02:23:32.370 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.138Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T02:23:32.371 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.138Z caller=main.go:947 level=info msg="TSDB started" 2024-04-06T02:23:32.371 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.138Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T02:23:32.371 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.157Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.651792ms db_storage=761ns remote_storage=1.457µs web_handler=320ns query_engine=698ns scrape=570.769µs scrape_sd=99.198µs notify=12.836µs notify_sd=9.484µs rules=18.707889ms 2024-04-06T02:23:32.371 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:32.157Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:32 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:23:32.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:23:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:32 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:33.966 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:33 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.966 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:33 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.966 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:33 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:23:33.966 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:33 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:33.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:23:33.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:33 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[24844]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[24844]: Upgrade: Updating mgr.x 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[24844]: Deploying daemon mgr.x on smithi104 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[29567]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[29567]: Upgrade: Updating mgr.x 2024-04-06T02:23:34.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:34 smithi017 ceph-mon[29567]: Deploying daemon mgr.x on smithi104 2024-04-06T02:23:35.001 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:34 smithi104 ceph-mon[25739]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:35.001 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:34 smithi104 ceph-mon[25739]: Upgrade: Updating mgr.x 2024-04-06T02:23:35.001 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:34 smithi104 ceph-mon[25739]: Deploying daemon mgr.x on smithi104 2024-04-06T02:23:35.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:35 smithi104 systemd[1]: Stopping Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:35.885 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:35 smithi104 podman[54502]: 2024-04-06 02:23:35.556636639 +0000 UTC m=+0.152911506 container died 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T02:23:36.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 podman[54502]: 2024-04-06 02:23:36.037499083 +0000 UTC m=+0.633773948 container cleanup 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-06T02:23:36.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 bash[54502]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x 2024-04-06T02:23:36.649 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 podman[54516]: 2024-04-06 02:23:36.234879636 +0000 UTC m=+0.677968581 container remove 3a23271d7c9a25b6a454db12eb5388f018443d2c939cab17a4e52d58b40c2ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-06T02:23:36.649 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:23:36.649 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Failed with result 'exit-code'. 2024-04-06T02:23:36.650 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 systemd[1]: Stopped Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:36.650 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Consumed 27.902s CPU time. 2024-04-06T02:23:36.650 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 systemd[1]: Starting Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:36.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:36 smithi104 ceph-mon[25739]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:36.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:36 smithi104 ceph-mon[25739]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:36.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 podman[54645]: 2024-04-06 02:23:36.782771623 +0000 UTC m=+0.018151287 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:23:36.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:36 smithi104 podman[54645]: 2024-04-06 02:23:36.894427178 +0000 UTC m=+0.129806884 container create 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235) 2024-04-06T02:23:36.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:36 smithi017 ceph-mon[29567]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:36.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:36 smithi017 ceph-mon[29567]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:36.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:36 smithi017 ceph-mon[24844]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-04-06T02:23:36.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:36 smithi017 ceph-mon[24844]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:37.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 podman[54645]: 2024-04-06 02:23:37.140449909 +0000 UTC m=+0.375829574 container init 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-06T02:23:37.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 podman[54645]: 2024-04-06 02:23:37.143405871 +0000 UTC m=+0.378785543 container start 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235) 2024-04-06T02:23:37.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 ceph-mgr[54664]: -- 172.21.15.104:0/1593848246 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595015665a0 con 0x55950153ac00 2024-04-06T02:23:37.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 bash[54645]: 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da 2024-04-06T02:23:37.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 systemd[1]: Started Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:38.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:37.723+0000 7f105baa8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:23:38.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:37.826+0000 7f105baa8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:23:38.505 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.505 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:38.626 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.626 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.626 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:38.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:38.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:38.756 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:23:38.757 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:38.521+0000 7f105baa8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:23:38.757 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:38.621+0000 7f105baa8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:23:38.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:23:38] "GET /metrics HTTP/1.1" 200 38241 "" "Prometheus/2.33.4" 2024-04-06T02:23:39.080 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:38.772+0000 7f105baa8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:23:39.081 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:38.869+0000 7f105baa8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:23:39.081 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:38.957+0000 7f105baa8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:23:39.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-mon[25739]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T02:23:39.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:39.227+0000 7f105baa8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:23:39.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:39.342+0000 7f105baa8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:39.720 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:39.439+0000 7f105baa8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:23:39.721 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:39.553+0000 7f105baa8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:23:39.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:39 smithi017 ceph-mon[29567]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T02:23:39.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:39 smithi017 ceph-mon[24844]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T02:23:39.984 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:39.736+0000 7f105baa8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:23:41.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:40.748+0000 7f105baa8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:23:41.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:40.882+0000 7f105baa8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:23:41.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-mon[25739]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:41.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.598 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.598 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-04-06T02:23:41.598 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:41.598 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:41.598 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:41.271+0000 7f105baa8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:23:41.598 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:41.369+0000 7f105baa8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:23:41.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[29567]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:41.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[24844]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:41.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:41 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:41.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:41.613+0000 7f105baa8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:41.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:41.816+0000 7f105baa8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:23:41.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 podman[55369]: 2024-04-06 02:23:41.572629103 +0000 UTC m=+0.017642176 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:23:41.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 podman[55369]: 2024-04-06 02:23:41.689083921 +0000 UTC m=+0.134096993 container create 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , name=ubi8, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, release=236.1648460182, distribution-scope=public) 2024-04-06T02:23:42.403 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:42.145+0000 7f105baa8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:42.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 podman[55369]: 2024-04-06 02:23:41.983438708 +0000 UTC m=+0.428451783 container init 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, distribution-scope=public, name=ubi8, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=236.1648460182, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:23:42.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:41 smithi104 podman[55369]: 2024-04-06 02:23:41.986012943 +0000 UTC m=+0.431026025 container start 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, release=236.1648460182, name=ubi8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, version=8.5, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:23:42.403 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[55399]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:42.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[55399]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:42.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 bash[55369]: 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 2024-04-06T02:23:42.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:42.404 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 podman[55460]: 2024-04-06 02:23:42.10172925 +0000 UTC m=+0.076610156 container died 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, name=ubi8, com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, release=236.1648460182, io.buildah.version=1.24.2, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T02:23:42.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 podman[55460]: 2024-04-06 02:23:42.654467031 +0000 UTC m=+0.629347923 container remove 212410b0c7dd3551d4e6570464a3bc805e245826678fe63bdbb4133e20deb247 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vcs-type=git, name=ubi8, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:23:42.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:23:42.654 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:42 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:23:42.654 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:42.419+0000 7f105baa8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:42.654 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:42.670+0000 7f105baa8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:23:43.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:43 smithi104 ceph-mon[25739]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:43.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:43 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:43.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:43 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:43.574 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:43.541+0000 7f105baa8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[29567]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[24844]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:43 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:44.311 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:44.126+0000 7f105baa8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:23:44.311 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:44.227+0000 7f105baa8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:23:44.642 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:44.461+0000 7f105baa8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:23:44] ENGINE Bus STARTING 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: CherryPy Checker: 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: The Application mounted at '' has an empty config. 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:23:44] ENGINE Serving on http://:::9283 2024-04-06T02:23:44.643 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:23:44] ENGINE Bus STARTED 2024-04-06T02:23:45.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:45.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: Standby manager daemon x restarted 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: Standby manager daemon x restarted 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: Standby manager daemon x started 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:45.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:45.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:45.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:45 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: Standby manager daemon x restarted 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/342698284' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:23:45.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:45 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: mgrmap e32: y(active, since 30s), standbys: x 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:23:46.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:46 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-06T02:23:46.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: mgrmap e32: y(active, since 30s), standbys: x 2024-04-06T02:23:46.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:46.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:23:46.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: mgrmap e32: y(active, since 30s), standbys: x 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-06T02:23:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-06T02:23:46.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:23:46.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:46 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: Upgrade: Setting container_image for all mgr 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[24844]: Upgrade: It appears safe to stop mon.a 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:47.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T02:23:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: Upgrade: Setting container_image for all mgr 2024-04-06T02:23:47.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:47 smithi017 ceph-mon[29567]: Upgrade: It appears safe to stop mon.a 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: Upgrade: Setting container_image for all mgr 2024-04-06T02:23:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:47 smithi104 ceph-mon[25739]: Upgrade: It appears safe to stop mon.a 2024-04-06T02:23:48.880 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:23:48] "GET /metrics HTTP/1.1" 200 38243 "" "Prometheus/2.33.4" 2024-04-06T02:23:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:23:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:23:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: Upgrade: Updating mon.a 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: Deploying daemon mon.a on smithi017 2024-04-06T02:23:49.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[29567]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: Upgrade: Updating mon.a 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: Deploying daemon mon.a on smithi017 2024-04-06T02:23:49.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 ceph-mon[24844]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: Upgrade: Updating mon.a 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='mgr.y' 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: from='mgr.15279 172.21.15.17:0/3864138440' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: Deploying daemon mon.a on smithi017 2024-04-06T02:23:49.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:49 smithi104 ceph-mon[25739]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:50.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:49 smithi017 systemd[1]: Stopping Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:50.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[24840]: 2024-04-06T02:23:50.087+0000 7f3e5a51c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:23:50.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[24840]: 2024-04-06T02:23:50.087+0000 7f3e5a51c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-06T02:23:50.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 podman[64295]: 2024-04-06 02:23:50.140747569 +0000 UTC m=+0.163028452 container died 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, release=754, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T02:23:50.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 podman[64295]: 2024-04-06 02:23:50.504826167 +0000 UTC m=+0.527107058 container cleanup 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_CLEAN=True, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-06T02:23:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 bash[64295]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a 2024-04-06T02:23:51.114 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 podman[64307]: 2024-04-06 02:23:50.797724164 +0000 UTC m=+0.653486003 container remove 9d54ed275a9f245cfc6a5588afb5b75c4f17245fb5c17bda2e0d86ffbbe86eba (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , name=centos-stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, ceph=True) 2024-04-06T02:23:51.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-06T02:23:51.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 systemd[1]: Stopped Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:51.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:50 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service: Consumed 13.986s CPU time. 2024-04-06T02:23:51.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 systemd[1]: Starting Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 podman[64440]: 2024-04-06 02:23:51.247271821 +0000 UTC m=+0.018530324 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:23:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 podman[64440]: 2024-04-06 02:23:51.353715779 +0000 UTC m=+0.124974309 container create 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:23:51.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 podman[64440]: 2024-04-06 02:23:51.726764413 +0000 UTC m=+0.498022902 container init 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:23:51.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 podman[64440]: 2024-04-06 02:23:51.729924462 +0000 UTC m=+0.501182955 container start 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-06T02:23:51.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 2 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: pidfile_write: ignore empty --pid-file 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: load: jerasure load: lrc 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: RocksDB version: 7.9.2 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Git sha 0 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: DB SUMMARY 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: DB Session ID: 53AWIIQ0HCQK9LN80FZH 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3584644 ; 2024-04-06T02:23:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.env: 0x55f177824c60 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.info_log: 0x55f178655ae0 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:23:51.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.statistics: (nil) 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.use_fsync: 0 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:23:51.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.db_log_dir: 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.wal_dir: 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:23:51.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.write_buffer_manager: 0x55f178628820 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.unordered_write: 0 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.row_cache: None 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.wal_filter: None 2024-04-06T02:23:51.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.wal_compression: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:23:51.994 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_open_files: -1 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:23:51.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Compression algorithms supported: 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kZSTD supported: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T02:23:51.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.merge_operator: 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_filter: None 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f178655be0) 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:23:51.997 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 4 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x55f178659090 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-04-06T02:23:51.998 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: bloomfilter 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 5 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: max_auto_readahead_size: 262144 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: prepopulate_block_cache: 0 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: initial_auto_readahead_size: 8192 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression: NoCompression 2024-04-06T02:23:51.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.num_levels: 7 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:23:52.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:23:52.001 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:23:52.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:23:52.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:23:52.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:23:52.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:23:52.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:23:52.003 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:23:52.004 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.ttl: 2592000 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T02:23:52.005 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12061, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f4ba718f-42c7-4d55-8d31-52af9659ddcc 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370231759307, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370231769160, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3304208, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12062, "largest_seqno": 13646, "table_properties": {"data_size": 3296645, "index_size": 4751, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1797, "raw_key_size": 16509, "raw_average_key_size": 23, "raw_value_size": 3281488, "raw_average_value_size": 4714, "num_data_blocks": 215, "num_entries": 696, "num_filter_entries": 696, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712370231, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f4ba718f-42c7-4d55-8d31-52af9659ddcc", "db_session_id": "53AWIIQ0HCQK9LN80FZH", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370231769265, "job": 1, "event": "recovery_finished"} 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f17874a000 2024-04-06T02:23:52.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: DB pointer 0x55f178734000 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 1/0 3.15 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-06T02:23:52.007 INFO:journalctl@ceph.mon.a.smithi017.stdout: L6 1/0 7.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 10.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 640.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 640.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 256.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 256.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache BinnedLRUCache@0x55f178659090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???) e3 preinit fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).mds e1 new map 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).mds e1 print_map 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-04-06T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 ceph-mon[64459]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 bash[64440]: 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 2024-04-06T02:23:52.010 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:51 smithi017 systemd[1]: Started Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:53.020 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:52 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-04-06T02:23:53.020 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:52 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:53.020 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:52 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:23:53.020 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:52 smithi104 podman[55899]: 2024-04-06 02:23:52.820298728 +0000 UTC m=+0.017619750 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:23:53.020 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:52 smithi104 podman[55899]: 2024-04-06 02:23:52.918653274 +0000 UTC m=+0.115974275 container create a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, build-date=2022-03-28T10:36:18.413762, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., version=8.5, vcs-type=git, distribution-scope=public, com.redhat.component=ubi8-container, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:23:53.260 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ignoring --setuser ceph since I am not root 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ignoring --setgroup ceph since I am not root 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mgr[60263]: -- 172.21.15.17:0/2876976243 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584306b25a0 con 0x558430686800 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: mon.a calling monitor election 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: monmap epoch 3 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:23:53.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: min_mon_release 17 (quincy) 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: election_strategy: 1 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: fsmap 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: mgrmap e32: y(active, since 38s), standbys: x 2024-04-06T02:23:53.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: daemon iscsi.foo.smithi017.kcchvr on smithi017 is in unknown state 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: from='mgr.15279 ' entity='' 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[29567]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: mon.a calling monitor election 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: monmap epoch 3 2024-04-06T02:23:53.263 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: min_mon_release 17 (quincy) 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: election_strategy: 1 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: fsmap 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: mgrmap e32: y(active, since 38s), standbys: x 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: daemon iscsi.foo.smithi017.kcchvr on smithi017 is in unknown state 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:23:53.264 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: from='mgr.15279 ' entity='' 2024-04-06T02:23:53.265 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-mon[64459]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-06T02:23:53.280 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: ignoring --setuser ceph since I am not root 2024-04-06T02:23:53.280 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: ignoring --setgroup ceph since I am not root 2024-04-06T02:23:53.280 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mgr[54664]: -- 172.21.15.104:0/564780771 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df242ca5a0 con 0x55df2429e800 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: mon.a calling monitor election 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: monmap epoch 3 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: min_mon_release 17 (quincy) 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: election_strategy: 1 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:23:53.281 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: fsmap 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: osdmap e104: 8 total, 8 up, 8 in 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: mgrmap e32: y(active, since 38s), standbys: x 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: daemon iscsi.foo.smithi017.kcchvr on smithi017 is in unknown state 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: from='mgr.15279 ' entity='' 2024-04-06T02:23:53.282 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-mon[25739]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-06T02:23:53.282 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 podman[55899]: 2024-04-06 02:23:53.196396922 +0000 UTC m=+0.393717921 container init a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, distribution-scope=public, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.buildah.version=1.24.2) 2024-04-06T02:23:53.282 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 podman[55899]: 2024-04-06 02:23:53.199107827 +0000 UTC m=+0.396428833 container start a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, release=236.1648460182, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.24.2, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:23:53.283 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[55914]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:53.283 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[55914]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:23:53.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 bash[55899]: a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 2024-04-06T02:23:53.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:23:53.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 podman[55930]: 2024-04-06 02:23:53.314713315 +0000 UTC m=+0.075273218 container died a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, architecture=x86_64, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.buildah.version=1.24.2, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-type=git, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.openshift.expose-services=, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:23:54.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:53.781+0000 7f7092e2e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:23:54.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:53 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:53.875+0000 7f7092e2e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:23:54.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 podman[55930]: 2024-04-06 02:23:53.922831941 +0000 UTC m=+0.683391842 container remove a13a4be2b0e1137983696a35cf546f9d97ace56a58241100d8f00902da195d81 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, description=Ceph Grafana Container, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base rhel8, release=236.1648460182, vcs-type=git, name=ubi8) 2024-04-06T02:23:54.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:23:54.162 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:23:53 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:23:54.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:53.837+0000 7fca3c595200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T02:23:54.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:53 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:53.934+0000 7fca3c595200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T02:23:54.835 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:54.520+0000 7f7092e2e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:23:54.835 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:54.613+0000 7f7092e2e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:23:54.835 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:54.755+0000 7f7092e2e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:23:54.835 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:54.851+0000 7f7092e2e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:23:54.937 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:54.615+0000 7fca3c595200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T02:23:54.937 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:54.711+0000 7fca3c595200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T02:23:54.937 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:54.854+0000 7fca3c595200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T02:23:54.937 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:54.953+0000 7fca3c595200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T02:23:55.161 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:54 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:54.933+0000 7f7092e2e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:23:55.192 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.038+0000 7fca3c595200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T02:23:55.472 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-mon[25739]: mgrmap e34: y(active, since 39s), standbys: x 2024-04-06T02:23:55.472 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:55.198+0000 7f7092e2e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:23:55.473 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:55.308+0000 7f7092e2e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:55.473 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:55.393+0000 7f7092e2e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:23:55.473 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:55.488+0000 7f7092e2e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:23:55.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-mon[29567]: mgrmap e34: y(active, since 39s), standbys: x 2024-04-06T02:23:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-mon[64459]: mgrmap e34: y(active, since 39s), standbys: x 2024-04-06T02:23:55.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.307+0000 7fca3c595200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T02:23:55.488 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.419+0000 7fca3c595200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:55.769 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.507+0000 7fca3c595200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T02:23:55.769 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.605+0000 7fca3c595200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T02:23:55.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:55.666+0000 7f7092e2e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:23:56.237 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:55 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:55.785+0000 7fca3c595200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T02:23:56.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:56.619+0000 7f7092e2e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:23:56.911 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:56.749+0000 7f7092e2e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:23:57.233 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:56.774+0000 7fca3c595200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T02:23:57.233 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:56 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:56.908+0000 7fca3c595200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T02:23:57.233 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:57.248+0000 7fca3c595200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:23:57.358 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:57.085+0000 7f7092e2e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T02:23:57.358 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:57.176+0000 7f7092e2e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:23:57.358 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:57.374+0000 7f7092e2e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:57.487 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:57.341+0000 7fca3c595200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T02:23:57.661 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:57.561+0000 7f7092e2e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:23:57.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:57.544+0000 7fca3c595200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:57.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:57 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:57.737+0000 7fca3c595200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T02:23:58.093 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:57 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:57.850+0000 7f7092e2e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:58.280 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:58.033+0000 7fca3c595200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T02:23:58.280 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:58.295+0000 7fca3c595200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:58.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:58.109+0000 7f7092e2e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T02:23:58.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:58.346+0000 7f7092e2e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:23:58.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:58.542+0000 7fca3c595200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T02:23:59.411 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:59 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:59.176+0000 7f7092e2e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:23:59.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:59.379+0000 7fca3c595200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T02:24:00.052 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:59 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:59.747+0000 7f7092e2e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:24:00.052 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:23:59 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:23:59.847+0000 7f7092e2e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:24:00.052 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:24:00.068+0000 7f7092e2e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:24:00.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:23:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:23:59.966+0000 7fca3c595200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T02:24:00.238 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:00.069+0000 7fca3c595200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T02:24:00.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: Standby manager daemon x restarted 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: Standby manager daemon x started 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[64459]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:24:00.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: Standby manager daemon x restarted 2024-04-06T02:24:00.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: Standby manager daemon x started 2024-04-06T02:24:00.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:24:00.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:24:00.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:24:00.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-mon[29567]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: Standby manager daemon x restarted 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: Standby manager daemon x started 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T02:24:00.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-mon[25739]: from='mgr.? 172.21.15.104:0/4144221754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:24:00] ENGINE Bus STARTING 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: CherryPy Checker: 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: The Application mounted at '' has an empty config. 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:24:00] ENGINE Serving on http://:::9283 2024-04-06T02:24:00.412 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:24:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x[54660]: [06/Apr/2024:02:24:00] ENGINE Bus STARTED 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:00.299+0000 7fca3c595200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:24:00] ENGINE Bus STARTING 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: CherryPy Checker: 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: The Application mounted at '' has an empty config. 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:24:00] ENGINE Serving on http://:::9283 2024-04-06T02:24:00.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:24:00] ENGINE Bus STARTED 2024-04-06T02:24:01.330 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:24:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: mgrmap e35: y(active, since 45s), standbys: x 2024-04-06T02:24:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: Active manager daemon y restarted 2024-04-06T02:24:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: Activating manager daemon y 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: mgrmap e36: y(active, starting, since 0.00733548s), standbys: x 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:24:01.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: Manager daemon y is now available 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:24:01.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:01 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:24:01.487 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:01.322+0000 7fc9f2743700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: mgrmap e35: y(active, since 45s), standbys: x 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: Active manager daemon y restarted 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: Activating manager daemon y 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: mgrmap e36: y(active, starting, since 0.00733548s), standbys: x 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:24:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: Manager daemon y is now available 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:24:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: mgrmap e35: y(active, since 45s), standbys: x 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: Active manager daemon y restarted 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: Activating manager daemon y 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: mgrmap e36: y(active, starting, since 0.00733548s), standbys: x 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T02:24:01.490 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: Manager daemon y is now available 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:24:01.491 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:01 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T02:24:02.009 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:24:02.009 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (98s) 42s ago 6m 13.6M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:24:02.009 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 20s ago 6m - - 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 unknown 42s ago 5m - - 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (25s) 20s ago 11m 65.6M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (61s) 42s ago 12m 475M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (12m) 42s ago 12m 57.2M 2048M 17.2.0 e1d6a67b021e 9d54ed275a9f 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (11m) 20s ago 11m 45.7M 2048M 17.2.0 e1d6a67b021e 1b96570856b2 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (11m) 42s ago 11m 42.6M 2048M 17.2.0 e1d6a67b021e f26130fe4327 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (107s) 42s ago 7m 9.89M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (79s) 20s ago 7m 17.2M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (10m) 42s ago 10m 53.7M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (10m) 42s ago 10m 56.1M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (9m) 42s ago 9m 48.9M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (9m) 42s ago 9m 54.9M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (9m) 20s ago 9m 55.3M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (8m) 20s ago 8m 54.2M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:24:02.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (8m) 20s ago 8m 52.2M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (7m) 20s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (30s) 20s ago 7m 45.7M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (5m) 42s ago 5m 72.1M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (5m) 20s ago 5m 72.2M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (5m) 42s ago 5m 74.8M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:24:02.011 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (5m) 20s ago 5m 76.6M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:24:02.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[29567]: [06/Apr/2024:02:24:00] ENGINE Bus STARTING 2024-04-06T02:24:02.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[29567]: [06/Apr/2024:02:24:00] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[29567]: [06/Apr/2024:02:24:00] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[29567]: [06/Apr/2024:02:24:00] ENGINE Bus STARTED 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[29567]: mgrmap e37: y(active, since 1.00879s), standbys: x 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[64459]: [06/Apr/2024:02:24:00] ENGINE Bus STARTING 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[64459]: [06/Apr/2024:02:24:00] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[64459]: [06/Apr/2024:02:24:00] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[64459]: [06/Apr/2024:02:24:00] ENGINE Bus STARTED 2024-04-06T02:24:02.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:02 smithi017 ceph-mon[64459]: mgrmap e37: y(active, since 1.00879s), standbys: x 2024-04-06T02:24:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:02 smithi104 ceph-mon[25739]: [06/Apr/2024:02:24:00] ENGINE Bus STARTING 2024-04-06T02:24:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:02 smithi104 ceph-mon[25739]: [06/Apr/2024:02:24:00] ENGINE Serving on https://172.21.15.17:7150 2024-04-06T02:24:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:02 smithi104 ceph-mon[25739]: [06/Apr/2024:02:24:00] ENGINE Serving on http://172.21.15.17:8765 2024-04-06T02:24:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:02 smithi104 ceph-mon[25739]: [06/Apr/2024:02:24:00] ENGINE Bus STARTED 2024-04-06T02:24:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:02 smithi104 ceph-mon[25739]: mgrmap e37: y(active, since 1.00879s), standbys: x 2024-04-06T02:24:02.447 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:02.448 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:24:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-04-06T02:24:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:24:02.449 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:24:02.449 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/23 daemons upgraded", 2024-04-06T02:24:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-04-06T02:24:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:24:02.793 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:24:03.233 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-06T02:24:03.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: Cluster is now healthy 2024-04-06T02:24:03.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:03 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/4118860298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: Cluster is now healthy 2024-04-06T02:24:03.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/4118860298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: Cluster is now healthy 2024-04-06T02:24:03.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:03 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/4118860298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:04.116 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 6. 2024-04-06T02:24:04.116 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:04.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[29567]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[29567]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[29567]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[29567]: from='client.? 172.21.15.17:0/3763787616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[64459]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[64459]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[64459]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-06T02:24:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:04 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3763787616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:04.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:04.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-mon[25739]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:04.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-mon[25739]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:04.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-mon[25739]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-06T02:24:04.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-mon[25739]: from='client.? 172.21.15.17:0/3763787616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:04.917 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 podman[56589]: 2024-04-06 02:24:04.583678051 +0000 UTC m=+0.017921747 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:24:04.918 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 podman[56589]: 2024-04-06 02:24:04.681605258 +0000 UTC m=+0.115848930 container create 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , io.buildah.version=1.24.2, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, version=8.5) 2024-04-06T02:24:04.918 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 podman[56589]: 2024-04-06 02:24:04.934409387 +0000 UTC m=+0.368653087 container init 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, architecture=x86_64, release=236.1648460182, distribution-scope=public, version=8.5, description=Ceph Grafana Container, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-type=git, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:24:05.320 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 podman[56589]: 2024-04-06 02:24:04.937388485 +0000 UTC m=+0.371632160 container start 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, maintainer=Paul Cuzner , release=236.1648460182, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5) 2024-04-06T02:24:05.320 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[56615]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:05.321 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:04 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[56615]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:05.321 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 bash[56589]: 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 2024-04-06T02:24:05.321 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:05.321 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 podman[56634]: 2024-04-06 02:24:05.042359326 +0000 UTC m=+0.071903387 container died 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8.5, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-06T02:24:05.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 podman[56634]: 2024-04-06 02:24:05.629203066 +0000 UTC m=+0.658747153 container remove 7d654d2d74a85a1f4bd35c682ffeb59ee0e612d1a31fa7a9ec9b416275076577 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, description=Ceph Grafana Container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, distribution-scope=public, version=8.5, io.openshift.expose-services=, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, release=236.1648460182) 2024-04-06T02:24:05.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:24:05.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:05 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:06 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:07 smithi104 ceph-mon[25739]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-06T02:24:07.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:07 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:07 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:07 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:07 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[29567]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-06T02:24:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[64459]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:08.803 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:08.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:08] "GET /metrics HTTP/1.1" 200 35446 "" "Prometheus/2.33.4" 2024-04-06T02:24:09.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:09 smithi017 ceph-mon[29567]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:24:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:09 smithi017 ceph-mon[64459]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:24:09.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:09 smithi104 ceph-mon[25739]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:24:11.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:11.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:11 smithi104 ceph-mon[25739]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:24:11.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:11.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[29567]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi017:/etc/ceph/ceph.conf 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:11 smithi017 ceph-mon[64459]: Updating smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:24:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[64459]: Upgrade: It appears safe to stop mon.c 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:12.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-06T02:24:12.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:12 smithi017 ceph-mon[29567]: Upgrade: It appears safe to stop mon.c 2024-04-06T02:24:12.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:12.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-06T02:24:12.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:12 smithi104 ceph-mon[25739]: Upgrade: It appears safe to stop mon.c 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: Upgrade: Updating mon.c 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:13.716 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[29567]: Deploying daemon mon.c on smithi017 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: Upgrade: Updating mon.c 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:13 smithi017 ceph-mon[64459]: Deploying daemon mon.c on smithi017 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: Upgrade: Updating mon.c 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:13.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:13 smithi104 ceph-mon[25739]: Deploying daemon mon.c on smithi017 2024-04-06T02:24:14.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:14 smithi017 ceph-mon[29567]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:14.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:14 smithi017 ceph-mon[64459]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:14.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:14 smithi104 ceph-mon[25739]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:15.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 systemd[1]: Stopping Ceph mon.c for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:15.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c[29563]: 2024-04-06T02:24:15.301+0000 7f53ab8da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:24:15.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c[29563]: 2024-04-06T02:24:15.301+0000 7f53ab8da700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-06T02:24:15.827 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 podman[66840]: 2024-04-06 02:24:15.556732214 +0000 UTC m=+0.348612318 container died f26130fe4327e613aa40c92c5c49e401a9580b8cec6506f5f5debc8cf542d57d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., name=centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-06T02:24:16.016 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:15 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 7. 2024-04-06T02:24:16.016 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:15 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:16.016 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:15 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:16.016 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:15 smithi104 podman[58002]: 2024-04-06 02:24:15.821683726 +0000 UTC m=+0.017381011 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:24:16.016 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:15 smithi104 podman[58002]: 2024-04-06 02:24:15.92037874 +0000 UTC m=+0.116076008 container create c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, release=236.1648460182, io.openshift.expose-services=, version=8.5, name=ubi8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container) 2024-04-06T02:24:16.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 podman[66840]: 2024-04-06 02:24:15.994621056 +0000 UTC m=+0.786501159 container cleanup f26130fe4327e613aa40c92c5c49e401a9580b8cec6506f5f5debc8cf542d57d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD) 2024-04-06T02:24:16.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:15 smithi017 bash[66840]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c 2024-04-06T02:24:16.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 podman[58002]: 2024-04-06 02:24:16.131051644 +0000 UTC m=+0.326748914 container init c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.buildah.version=1.24.2, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=236.1648460182, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vcs-type=git) 2024-04-06T02:24:16.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 podman[58002]: 2024-04-06 02:24:16.133785074 +0000 UTC m=+0.329482350 container start c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, io.openshift.expose-services=, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container) 2024-04-06T02:24:16.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58017]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:16.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58017]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:16.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 bash[58002]: c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 2024-04-06T02:24:16.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:16.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 podman[58033]: 2024-04-06 02:24:16.220959583 +0000 UTC m=+0.052719371 container died c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner ) 2024-04-06T02:24:16.600 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 podman[66855]: 2024-04-06 02:24:16.286559963 +0000 UTC m=+0.725024584 container remove f26130fe4327e613aa40c92c5c49e401a9580b8cec6506f5f5debc8cf542d57d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-06T02:24:16.600 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.c.service: Deactivated successfully. 2024-04-06T02:24:16.600 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 systemd[1]: Stopped Ceph mon.c for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:16.600 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.c.service: Consumed 7.801s CPU time. 2024-04-06T02:24:16.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 systemd[1]: Starting Ceph mon.c for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:16.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 podman[66983]: 2024-04-06 02:24:16.821425481 +0000 UTC m=+0.108656689 container create 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:24:16.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:16 smithi017 podman[66983]: 2024-04-06 02:24:16.730959904 +0000 UTC m=+0.018191071 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:24:17.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 podman[58033]: 2024-04-06 02:24:16.776501738 +0000 UTC m=+0.608261525 container remove c1b06bc000466ba1d6034d30707f052e98ea1de3126b1a1a059ce48c33b6d690 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8, io.openshift.expose-services=, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:24:17.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:24:17.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:16 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:24:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 podman[66983]: 2024-04-06 02:24:17.023962312 +0000 UTC m=+0.311193486 container init 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-06T02:24:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 podman[66983]: 2024-04-06 02:24:17.027449181 +0000 UTC m=+0.314680350 container start 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-06T02:24:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:24:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 2 2024-04-06T02:24:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: pidfile_write: ignore empty --pid-file 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: load: jerasure load: lrc 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: RocksDB version: 7.9.2 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Git sha 0 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: DB SUMMARY 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: DB Session ID: W2C6P0U2Y1XLJT14G6BD 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 566213 ; 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:24:17.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.env: 0x555a12ce6c60 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.info_log: 0x555a13745ae0 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:24:17.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.statistics: (nil) 2024-04-06T02:24:17.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.use_fsync: 0 2024-04-06T02:24:17.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:24:17.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:24:17.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.db_log_dir: 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.wal_dir: 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:24:17.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.write_buffer_manager: 0x555a13718820 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.unordered_write: 0 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:24:17.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.row_cache: None 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.wal_filter: None 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.wal_compression: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:24:17.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:24:17.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_open_files: -1 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Compression algorithms supported: 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kZSTD supported: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:24:17.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.merge_operator: 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_filter: None 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:24:17.247 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555a13745be0) 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_type: 0 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_index_type: 0 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_shortening: 1 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: checksum: 4 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: no_block_cache: 0 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache: 0x555a13749090 2024-04-06T02:24:17.248 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_options: 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: capacity : 536870912 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_shard_bits : 4 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: strict_capacity_limit : 0 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_compressed: (nil) 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: persistent_cache: (nil) 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size: 4096 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size_deviation: 10 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_restart_interval: 16 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_block_restart_interval: 1 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: metadata_block_size: 4096 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: partition_filters: 0 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: use_delta_encoding: 1 2024-04-06T02:24:17.249 INFO:journalctl@ceph.mon.c.smithi017.stdout: filter_policy: bloomfilter 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: whole_key_filtering: 1 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: verify_compression: 0 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: format_version: 5 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_index_compression: 1 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_align: 0 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: max_auto_readahead_size: 262144 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: prepopulate_block_cache: 0 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: initial_auto_readahead_size: 8192 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression: NoCompression 2024-04-06T02:24:17.250 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.num_levels: 7 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:24:17.251 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:24:17.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T02:24:17.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:24:17.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.ttl: 2592000 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:24:17.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:24:17.256 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15084, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e31d71f9-6afd-452b-99d0-92f237ae68bd 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370257056591, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370257058673, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 350055, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15089, "largest_seqno": 15290, "table_properties": {"data_size": 348021, "index_size": 766, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2461, "raw_average_key_size": 25, "raw_value_size": 345620, "raw_average_value_size": 3526, "num_data_blocks": 33, "num_entries": 98, "num_filter_entries": 98, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712370257, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e31d71f9-6afd-452b-99d0-92f237ae68bd", "db_session_id": "W2C6P0U2Y1XLJT14G6BD", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370257058758, "job": 1, "event": "recovery_finished"} 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555a1383a000 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: DB pointer 0x555a13824000 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T02:24:17.257 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** DB Stats ** 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: L0 1/0 341.85 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 267.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: L6 1/0 11.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T02:24:17.258 INFO:journalctl@ceph.mon.c.smithi017.stdout: Sum 2/0 11.83 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 267.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 267.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 267.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:24:17.259 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative compaction: 0.00 GB write, 80.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval compaction: 0.00 GB write, 80.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache BinnedLRUCache@0x555a13749090#2 capacity: 512.00 MB usage: 1.14 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.83 KB,0.000157952%) Misc(1,0.00 KB,0%) 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: starting mon.c rank 1 at public addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] at bind addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???) e3 preinit fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).mds e1 new map 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).mds e1 print_map 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: e1 2024-04-06T02:24:17.260 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout: legacy client fscid: -1 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout: No filesystems configured 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 ceph-mon[67002]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 bash[66983]: 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 2024-04-06T02:24:17.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:17 smithi017 systemd[1]: Started Ceph mon.c for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:18.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:18 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:18.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:18 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:18.738 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:18] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.33.4" 2024-04-06T02:24:18.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:18 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:18.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: mon.c calling monitor election 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: mon.a calling monitor election 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: monmap epoch 3 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: min_mon_release 17 (quincy) 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: election_strategy: 1 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: fsmap 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: mgrmap e39: y(active, since 18s), standbys: x 2024-04-06T02:24:19.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:19.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:19.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:19.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:19 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:19.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:19.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: mon.c calling monitor election 2024-04-06T02:24:19.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:19.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: mon.a calling monitor election 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: monmap epoch 3 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: min_mon_release 17 (quincy) 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: election_strategy: 1 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: fsmap 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:19.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: mgrmap e39: y(active, since 18s), standbys: x 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: mon.c calling monitor election 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: mon.a calling monitor election 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: monmap epoch 3 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: last_changed 2024-04-06T02:12:45.851852+0000 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: min_mon_release 17 (quincy) 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: election_strategy: 1 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: fsmap 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:19.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: mgrmap e39: y(active, since 18s), standbys: x 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:19.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:20.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:20 smithi017 ceph-mon[67002]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-06T02:24:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:20 smithi017 ceph-mon[64459]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-06T02:24:20.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:20 smithi104 ceph-mon[25739]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-06T02:24:22.487 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:22 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:22.067+0000 7fc9f2743700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-06T02:24:23.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:23 smithi104 ceph-mon[25739]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:23.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:23 smithi104 ceph-mon[25739]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:23.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:23 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:23.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:23 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[67002]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[64459]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:23 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:25 smithi104 ceph-mon[25739]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:25.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:25 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:25 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[64459]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:25.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[67002]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:25.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:25.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:25 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:27.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:27 smithi104 ceph-mon[25739]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:27.168 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:26 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 8. 2024-04-06T02:24:27.168 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:26 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:27.168 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:26 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:27.168 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 podman[58114]: 2024-04-06 02:24:27.062819459 +0000 UTC m=+0.017320681 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:24:27.168 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 podman[58114]: 2024-04-06 02:24:27.185025504 +0000 UTC m=+0.139526767 container create c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, distribution-scope=public, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, name=ubi8, io.openshift.expose-services=, vcs-type=git, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:24:27.437 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:27 smithi017 ceph-mon[67002]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:27.438 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:27 smithi017 ceph-mon[64459]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 podman[58114]: 2024-04-06 02:24:27.554012091 +0000 UTC m=+0.508513314 container init c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, version=8.5, description=Ceph Grafana Container, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8) 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 podman[58114]: 2024-04-06 02:24:27.556421204 +0000 UTC m=+0.510922419 container start c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, architecture=x86_64, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=) 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58128]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58128]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 bash[58114]: c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:27.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:27 smithi104 podman[58205]: 2024-04-06 02:24:27.6880681 +0000 UTC m=+0.097618542 container died c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, io.buildah.version=1.24.2, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., name=ubi8, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.openshift.expose-services=, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container) 2024-04-06T02:24:28.552 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:28 smithi104 podman[58205]: 2024-04-06 02:24:28.282874545 +0000 UTC m=+0.692425039 container remove c3a64d52ad2660acab185d832a7ea55c023b7123822df7d3817543ba798715c2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, distribution-scope=public, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner ) 2024-04-06T02:24:28.552 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:28 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:24:28.552 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:28 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:24:28.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-06T02:24:28.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-mon[25739]: Upgrade: It appears safe to stop mon.b 2024-04-06T02:24:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:28] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.33.4" 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:28.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[67002]: Upgrade: It appears safe to stop mon.b 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-06T02:24:28.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:28 smithi017 ceph-mon[64459]: Upgrade: It appears safe to stop mon.b 2024-04-06T02:24:28.815 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:30.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: Upgrade: Updating mon.b 2024-04-06T02:24:30.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:30.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: Deploying daemon mon.b on smithi104 2024-04-06T02:24:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:30 smithi104 ceph-mon[25739]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:30.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: Upgrade: Updating mon.b 2024-04-06T02:24:30.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: Deploying daemon mon.b on smithi104 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[67002]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: Upgrade: Updating mon.b 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: Deploying daemon mon.b on smithi104 2024-04-06T02:24:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:30 smithi017 ceph-mon[64459]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:24:31.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:31 smithi104 systemd[1]: Stopping Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b[25720]: 2024-04-06T02:24:31.389+0000 7f82a1141700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:24:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b[25720]: 2024-04-06T02:24:31.389+0000 7f82a1141700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-06T02:24:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:31 smithi104 podman[58506]: 2024-04-06 02:24:31.635234728 +0000 UTC m=+0.367267763 container died 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., version=8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8) 2024-04-06T02:24:32.241 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 podman[58506]: 2024-04-06 02:24:32.048445656 +0000 UTC m=+0.780478689 container cleanup 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-06T02:24:32.241 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 bash[58506]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b 2024-04-06T02:24:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 podman[58520]: 2024-04-06 02:24:32.257878509 +0000 UTC m=+0.613949966 container remove 1b96570856b2d2853dfab6f04319899cc1e5a531844d26855dc649d24ac720ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_CLEAN=True) 2024-04-06T02:24:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.b.service: Deactivated successfully. 2024-04-06T02:24:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 systemd[1]: Stopped Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.b.service: Consumed 9.358s CPU time. 2024-04-06T02:24:33.141 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:32 smithi104 systemd[1]: Starting Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:33.141 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 podman[58649]: 2024-04-06 02:24:32.945274211 +0000 UTC m=+0.017809714 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:24:33.141 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 podman[58649]: 2024-04-06 02:24:33.050396386 +0000 UTC m=+0.122931890 container create bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-06T02:24:33.603 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:24:33.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 podman[58649]: 2024-04-06 02:24:33.293309775 +0000 UTC m=+0.365845279 container init bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-06T02:24:33.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 podman[58649]: 2024-04-06 02:24:33.296353629 +0000 UTC m=+0.368889141 container start bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 2 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: pidfile_write: ignore empty --pid-file 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: load: jerasure load: lrc 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: RocksDB version: 7.9.2 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Git sha 0 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: DB SUMMARY 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: DB Session ID: NMF38TOBR2I94PF51OCG 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-04-06T02:24:33.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1239076 ; 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.env: 0x56295974dc60 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.info_log: 0x56295a71bae0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.statistics: (nil) 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.use_fsync: 0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:24:33.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.db_log_dir: 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.wal_dir: 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:24:33.664 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.write_buffer_manager: 0x56295a6ee820 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:24:33.665 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.unordered_write: 0 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.row_cache: None 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.wal_filter: None 2024-04-06T02:24:33.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:24:33.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:24:33.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:24:33.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:24:33.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.wal_compression: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:24:33.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_open_files: -1 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:24:33.669 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Compression algorithms supported: 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kZSTD supported: 0 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.merge_operator: 2024-04-06T02:24:33.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_filter: None 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56295a71bbe0) 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_type: 0 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_index_type: 0 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_shortening: 1 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:24:33.671 INFO:journalctl@ceph.mon.b.smithi104.stdout: checksum: 4 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: no_block_cache: 0 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache: 0x56295a71f090 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_options: 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: capacity : 536870912 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_shard_bits : 4 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: strict_capacity_limit : 0 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_compressed: (nil) 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: persistent_cache: (nil) 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size: 4096 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size_deviation: 10 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_restart_interval: 16 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_block_restart_interval: 1 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: metadata_block_size: 4096 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: partition_filters: 0 2024-04-06T02:24:33.672 INFO:journalctl@ceph.mon.b.smithi104.stdout: use_delta_encoding: 1 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: filter_policy: bloomfilter 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: whole_key_filtering: 1 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: verify_compression: 0 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: format_version: 5 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_index_compression: 1 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_align: 0 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: max_auto_readahead_size: 262144 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: prepopulate_block_cache: 0 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: initial_auto_readahead_size: 8192 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression: NoCompression 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:24:33.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.num_levels: 7 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:24:33.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:24:33.675 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:24:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:24:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.ttl: 2592000 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T02:24:33.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14986, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-06T02:24:33.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f44be719-33cf-4c24-a058-097ef520992b 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370273325363, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370273329032, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 735212, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14991, "largest_seqno": 15710, "table_properties": {"data_size": 731518, "index_size": 1463, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 9409, "raw_average_key_size": 20, "raw_value_size": 723511, "raw_average_value_size": 1566, "num_data_blocks": 66, "num_entries": 462, "num_filter_entries": 462, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712370273, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f44be719-33cf-4c24-a058-097ef520992b", "db_session_id": "NMF38TOBR2I94PF51OCG", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712370273329121, "job": 1, "event": "recovery_finished"} 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56295a810000 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: DB pointer 0x56295a7fa000 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** DB Stats ** 2024-04-06T02:24:33.680 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 717.98 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 367.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: L6 1/0 11.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 2/0 12.19 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 367.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 367.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.681 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 367.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:24:33.682 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 86.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 86.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache BinnedLRUCache@0x56295a71f090#2 capacity: 512.00 MB usage: 2.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: starting mon.b rank 2 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???) e3 preinit fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).mds e1 new map 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).mds e1 print_map 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: e1 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: legacy client fscid: -1 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-04-06T02:24:33.683 INFO:journalctl@ceph.mon.b.smithi104.stdout: No filesystems configured 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 ceph-mon[58668]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 bash[58649]: bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c 2024-04-06T02:24:33.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:33 smithi104 systemd[1]: Started Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:38.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:38.678 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 9. 2024-04-06T02:24:38.678 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:38.678 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:38.679 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 podman[58762]: 2024-04-06 02:24:38.567234751 +0000 UTC m=+0.017654861 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:24:38.942 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:38] "GET /metrics HTTP/1.1" 200 38262 "" "Prometheus/2.33.4" 2024-04-06T02:24:38.947 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 podman[58762]: 2024-04-06 02:24:38.695076258 +0000 UTC m=+0.145496368 container create 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, maintainer=Paul Cuzner , name=ubi8, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, version=8.5, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:24:39.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:38 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:39.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:38 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 podman[58762]: 2024-04-06 02:24:38.964337635 +0000 UTC m=+0.414757746 container init 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, distribution-scope=public, release=236.1648460182, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-type=git, maintainer=Paul Cuzner ) 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 podman[58762]: 2024-04-06 02:24:38.966967536 +0000 UTC m=+0.417387655 container start 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, distribution-scope=public, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.24.2, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58776]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[58776]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 bash[58762]: 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:39.411 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 podman[58792]: 2024-04-06 02:24:39.071974727 +0000 UTC m=+0.071927147 container died 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, distribution-scope=public, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.expose-services=, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-06T02:24:39.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 podman[58792]: 2024-04-06 02:24:39.651030014 +0000 UTC m=+0.650982435 container remove 56d2788f827fa553a7857f0e081e326193e0d6c094a4738133a952657b7d91ed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182) 2024-04-06T02:24:39.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:24:39.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:39 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:24:40.270 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:39 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 17s ago 7m 17.6M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 34s ago 6m - - 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (75s) 17s ago 6m 49.5M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (63s) 34s ago 11m 447M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (99s) 17s ago 13m 492M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (49s) 17s ago 13m 41.6M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:24:40.280 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 starting - - - 2048M 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (23s) 17s ago 12m 17.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 17s ago 7m 21.8M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (117s) 34s ago 7m 18.1M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (11m) 17s ago 11m 53.6M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (10m) 17s ago 10m 57.0M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (10m) 17s ago 10m 49.4M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (10m) 17s ago 10m 55.6M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (9m) 34s ago 9m 55.5M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (9m) 34s ago 9m 54.3M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (9m) 34s ago 8m 52.5M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (8m) 34s ago 8m 54.1M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (69s) 34s ago 7m 57.5M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (6m) 17s ago 6m 73.4M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (6m) 34s ago 6m 72.7M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:24:40.281 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (6m) 17s ago 6m 76.0M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:24:40.282 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (6m) 34s ago 6m 76.9M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:24:40.687 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:24:40.687 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:24:40.687 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:24:40.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:24:40.688 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:24:41.031 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:24:41.031 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "4/23 daemons upgraded", 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:24:41.032 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: mon.b calling monitor election 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: mon.c calling monitor election 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: mon.a calling monitor election 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:41.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: monmap epoch 4 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: last_changed 2024-04-06T02:24:39.989239+0000 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: min_mon_release 19 (squid) 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: election_strategy: 1 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: fsmap 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: mgrmap e39: y(active, since 39s), standbys: x 2024-04-06T02:24:41.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:41.163 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:41 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2115876064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: mon.b calling monitor election 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: mon.c calling monitor election 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: mon.a calling monitor election 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: monmap epoch 4 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: last_changed 2024-04-06T02:24:39.989239+0000 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: min_mon_release 19 (squid) 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: election_strategy: 1 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: fsmap 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: mgrmap e39: y(active, since 39s), standbys: x 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2115876064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: mon.b calling monitor election 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: mon.c calling monitor election 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: mon.a calling monitor election 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: monmap epoch 4 2024-04-06T02:24:41.240 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: last_changed 2024-04-06T02:24:39.989239+0000 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: created 2024-04-06T02:11:16.126688+0000 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: min_mon_release 19 (squid) 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: election_strategy: 1 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: 1: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: fsmap 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: osdmap e105: 8 total, 8 up, 8 in 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: mgrmap e39: y(active, since 39s), standbys: x 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:41.241 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:41 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2115876064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:24:41.446 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:24:41.446 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:24:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:24:42.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:42 smithi104 ceph-mon[58668]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:42 smithi104 ceph-mon[58668]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:42 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1077755024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:42.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[67002]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[67002]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1077755024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:42.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[64459]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[64459]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:24:42.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:42 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1077755024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:24:43.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:43 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:43.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:43 smithi017 ceph-mon[67002]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:43.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:43 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:43.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:43 smithi017 ceph-mon[64459]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:43.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:43 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:43.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:43 smithi104 ceph-mon[58668]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:43.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:24:43.339+0000 7fc9f2743700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-06T02:24:44.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:44 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:44.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:44 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:44.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:44 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:44.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:44 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:44.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:44 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:44.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:44 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:45 smithi104 ceph-mon[58668]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:45.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[67002]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:45.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[64459]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:46.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:46 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:46.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:46 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:46 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:24:47.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:47 smithi104 ceph-mon[58668]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:47.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:47 smithi017 ceph-mon[67002]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:47.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:47 smithi017 ceph-mon[64459]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:24:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: Reconfiguring mon.a (monmap changed)... 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:48 smithi104 ceph-mon[58668]: Reconfiguring daemon mon.a on smithi017 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:48] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.33.4" 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: Reconfiguring mon.a (monmap changed)... 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[64459]: Reconfiguring daemon mon.a on smithi017 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:48.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:48.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: Reconfiguring mon.a (monmap changed)... 2024-04-06T02:24:48.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:48.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:48.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:48.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:48 smithi017 ceph-mon[67002]: Reconfiguring daemon mon.a on smithi017 2024-04-06T02:24:49.912 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:49 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 10. 2024-04-06T02:24:49.912 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:49 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:49.912 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:49 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:24:49.912 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:49 smithi104 podman[59836]: 2024-04-06 02:24:49.810248703 +0000 UTC m=+0.017546649 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:24:50.197 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:49 smithi104 podman[59836]: 2024-04-06 02:24:49.929159057 +0000 UTC m=+0.136456975 container create 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.openshift.expose-services=, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, architecture=x86_64, name=ubi8, version=8.5) 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 podman[59836]: 2024-04-06 02:24:50.214998291 +0000 UTC m=+0.422296208 container init 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , architecture=x86_64, name=ubi8, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 podman[59836]: 2024-04-06 02:24:50.218084098 +0000 UTC m=+0.425382023 container start 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc.) 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[59852]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[59852]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 bash[59836]: 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 2024-04-06T02:24:50.463 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:24:50.464 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 podman[59868]: 2024-04-06 02:24:50.303086981 +0000 UTC m=+0.052096901 container died 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vendor=Red Hat, Inc., version=8.5, name=ubi8, io.openshift.expose-services=, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, maintainer=Paul Cuzner , distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:24:50.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: Reconfiguring mgr.y (monmap changed)... 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[64459]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: Reconfiguring mgr.y (monmap changed)... 2024-04-06T02:24:50.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:24:50.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:24:50.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:50.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:24:50.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:50 smithi017 ceph-mon[67002]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: Reconfiguring mgr.y (monmap changed)... 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: Reconfiguring daemon mgr.y on smithi017 2024-04-06T02:24:50.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:50 smithi104 ceph-mon[58668]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:51.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 podman[59868]: 2024-04-06 02:24:50.885072128 +0000 UTC m=+0.634082027 container remove 6f85726acadb6bec97513c763cb0d1a4f7a92045cc0efbfa8c05808b9e257b86 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.openshift.expose-services=, maintainer=Paul Cuzner , version=8.5, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc.) 2024-04-06T02:24:51.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:24:51.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:24:50 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:52.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[67002]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:52.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:52 smithi017 ceph-mon[64459]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: Reconfiguring mon.c (monmap changed)... 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:52 smithi104 ceph-mon[58668]: Reconfiguring daemon mon.c on smithi017 2024-04-06T02:24:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:24:53.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:24:53.427 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:24:53.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:54.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:54 smithi017 ceph-mon[67002]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T02:24:54.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:54 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.0 on smithi017 2024-04-06T02:24:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:54 smithi017 ceph-mon[64459]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T02:24:54.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:54 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.0 on smithi017 2024-04-06T02:24:54.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:54 smithi104 ceph-mon[58668]: Reconfiguring osd.0 (monmap changed)... 2024-04-06T02:24:54.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:54 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.0 on smithi017 2024-04-06T02:24:55.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[67002]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:55.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:24:55.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:55.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[64459]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:55.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:24:55.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:55 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:55.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:55 smithi104 ceph-mon[58668]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:24:55.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:55 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:55 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:55.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:55 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:24:55.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:55 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:56.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:56 smithi104 ceph-mon[58668]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T02:24:56.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:56 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.1 on smithi017 2024-04-06T02:24:56.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:56 smithi017 ceph-mon[67002]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T02:24:56.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:56 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.1 on smithi017 2024-04-06T02:24:56.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:56 smithi017 ceph-mon[64459]: Reconfiguring osd.1 (monmap changed)... 2024-04-06T02:24:56.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:56 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.1 on smithi017 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.2 on smithi017 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T02:24:57.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:24:57.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:57.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:57 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.2 on smithi017 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: Reconfiguring osd.2 (monmap changed)... 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:57.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:57 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.2 on smithi017 2024-04-06T02:24:58.661 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:24:58] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.33.4" 2024-04-06T02:24:58.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:24:58.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T02:24:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:24:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:58.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.3 on smithi017 2024-04-06T02:24:58.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:24:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:24:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:24:58.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:24:58.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.3 on smithi017 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:24:58.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: Reconfiguring osd.3 (monmap changed)... 2024-04-06T02:24:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:24:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:24:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:24:58 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.3 on smithi017 2024-04-06T02:25:00.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: Reconfiguring rgw.foo.smithi017.ffhtpi (monmap changed)... 2024-04-06T02:25:00.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: Reconfiguring daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: Reconfiguring rgw.foo.smithi017.ffhtpi (monmap changed)... 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: Reconfiguring daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:00.719 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: Reconfiguring rgw.foo.smithi017.ffhtpi (monmap changed)... 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: Reconfiguring daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:00.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:01.176 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:00 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 11. 2024-04-06T02:25:01.176 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:00 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:01.176 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:00 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:01.176 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 podman[59938]: 2024-04-06 02:25:01.069132168 +0000 UTC m=+0.017758414 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:25:01.428 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 podman[59938]: 2024-04-06 02:25:01.193499981 +0000 UTC m=+0.142126209 container create f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, release=236.1648460182, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , vcs-type=git, io.openshift.tags=base rhel8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, description=Ceph Grafana Container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 podman[59938]: 2024-04-06 02:25:01.445604602 +0000 UTC m=+0.394230835 container init f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , architecture=x86_64, build-date=2022-03-28T10:36:18.413762, version=8.5, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vcs-type=git) 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 podman[59938]: 2024-04-06 02:25:01.448380001 +0000 UTC m=+0.397006239 container start f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, vcs-type=git, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.expose-services=) 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[59952]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[59952]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 bash[59938]: f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:01.911 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:01 smithi104 podman[59967]: 2024-04-06 02:25:01.579538042 +0000 UTC m=+0.097573146 container died f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, vcs-type=git, maintainer=Paul Cuzner , io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., architecture=x86_64, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.openshift.expose-services=) 2024-04-06T02:25:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: Reconfiguring rgw.smpl.smithi017.qqrgsv (monmap changed)... 2024-04-06T02:25:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:02.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:02 smithi104 ceph-mon[58668]: Reconfiguring daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:25:02.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:02 smithi104 podman[59967]: 2024-04-06 02:25:02.140651206 +0000 UTC m=+0.658686313 container remove f941ba607f3bf19137b7e176076049c92b2563e5fb64abe4b19306f5d0ab26dd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, distribution-scope=public, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-06T02:25:02.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:02 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:25:02.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:02 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: Reconfiguring rgw.smpl.smithi017.qqrgsv (monmap changed)... 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[64459]: Reconfiguring daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: Reconfiguring rgw.smpl.smithi017.qqrgsv (monmap changed)... 2024-04-06T02:25:02.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:02.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:02.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:02 smithi017 ceph-mon[67002]: Reconfiguring daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:25:03.407 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:25:03.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:25:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:04.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:04 smithi104 ceph-mon[58668]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:25:04.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:04 smithi104 ceph-mon[58668]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:25:04.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:04 smithi017 ceph-mon[64459]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:25:04.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:04 smithi017 ceph-mon[64459]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:25:04.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:04 smithi017 ceph-mon[67002]: Reconfiguring mon.b (monmap changed)... 2024-04-06T02:25:04.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:04 smithi017 ceph-mon[67002]: Reconfiguring daemon mon.b on smithi104 2024-04-06T02:25:05.366 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:05.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:25:05.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:25:05.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:05 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:05.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:05.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:25:05.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:05 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:06.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: Reconfiguring mgr.x (monmap changed)... 2024-04-06T02:25:06.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: Reconfiguring daemon mgr.x on smithi104 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: Reconfiguring mgr.x (monmap changed)... 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: Reconfiguring daemon mgr.x on smithi104 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:25:06.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: Reconfiguring mgr.x (monmap changed)... 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: Reconfiguring daemon mgr.x on smithi104 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:25:06.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:07.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:07 smithi104 ceph-mon[58668]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:07.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:07 smithi104 ceph-mon[58668]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T02:25:07.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:07 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.4 on smithi104 2024-04-06T02:25:07.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[67002]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:07.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[67002]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T02:25:07.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.4 on smithi104 2024-04-06T02:25:07.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[64459]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:07.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[64459]: Reconfiguring osd.4 (monmap changed)... 2024-04-06T02:25:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:07 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.4 on smithi104 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.5 on smithi104 2024-04-06T02:25:08.912 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:08.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:08] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.33.4" 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.5 on smithi104 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: Reconfiguring osd.5 (monmap changed)... 2024-04-06T02:25:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:25:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:08 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.5 on smithi104 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: Reconfiguring osd.6 (monmap changed)... 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.6 on smithi104 2024-04-06T02:25:10.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:10 smithi104 ceph-mon[58668]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:10.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: Reconfiguring osd.6 (monmap changed)... 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.6 on smithi104 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[67002]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: Reconfiguring osd.6 (monmap changed)... 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:10.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.6 on smithi104 2024-04-06T02:25:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:10 smithi017 ceph-mon[64459]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:11.816 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: Reconfiguring osd.7 (monmap changed)... 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: Reconfiguring daemon osd.7 on smithi104 2024-04-06T02:25:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:12 smithi104 ceph-mon[58668]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:12 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 12. 2024-04-06T02:25:12.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:12 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:12.412 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:12 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:12.488 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 49s ago 7m 17.6M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 29s ago 7m - - 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (107s) 49s ago 6m 49.5M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (95s) 29s ago 12m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (2m) 49s ago 13m 492M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (81s) 49s ago 13m 41.6M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (39s) 29s ago 12m 16.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (55s) 49s ago 12m 17.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 49s ago 8m 21.8M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (2m) 29s ago 8m 17.7M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (11m) 49s ago 11m 53.6M 2737M 17.2.0 e1d6a67b021e a0d080c7a74c 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (11m) 49s ago 11m 57.0M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (11m) 49s ago 11m 49.4M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (10m) 49s ago 10m 55.6M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (10m) 29s ago 10m 55.7M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:25:12.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (9m) 29s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (9m) 29s ago 9m 53.0M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (9m) 29s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (101s) 29s ago 8m 65.0M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (6m) 49s ago 6m 73.4M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (6m) 29s ago 6m 73.0M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (6m) 49s ago 6m 76.0M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:25:12.490 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (6m) 29s ago 6m 77.3M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:25:12.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: Reconfiguring osd.7 (monmap changed)... 2024-04-06T02:25:12.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:25:12.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: Reconfiguring daemon osd.7 on smithi104 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[67002]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: Reconfiguring osd.7 (monmap changed)... 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:25:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: Reconfiguring daemon osd.7 on smithi104 2024-04-06T02:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:12 smithi017 ceph-mon[64459]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:12.899 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:25:12.899 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:25:12.899 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:25:12.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:25:12.900 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:25:13.122 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60804]: 2024-04-06 02:25:12.918551847 +0000 UTC m=+0.018009146 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:25:13.123 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60804]: 2024-04-06 02:25:13.040864009 +0000 UTC m=+0.140321280 container create 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.component=ubi8-container, release=236.1648460182, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8) 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "5/23 daemons upgraded", 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:25:13.245 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:25:13.376 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60804]: 2024-04-06 02:25:13.301810303 +0000 UTC m=+0.401267575 container init 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, release=236.1648460182, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, description=Ceph Grafana Container, vendor=Red Hat, Inc.) 2024-04-06T02:25:13.376 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60804]: 2024-04-06 02:25:13.30474289 +0000 UTC m=+0.404200170 container start 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, name=ubi8, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc., description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-06T02:25:13.376 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[60884]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:13.376 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[60884]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:13.654 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:25:13.655 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:25:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:25:13.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 bash[60804]: 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa 2024-04-06T02:25:13.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:13.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60900]: 2024-04-06 02:25:13.417211688 +0000 UTC m=+0.077446297 container died 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=Ceph Grafana Container, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, name=ubi8, io.buildah.version=1.24.2) 2024-04-06T02:25:13.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2049455059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: Reconfiguring rgw.foo.smithi104.iurqzi (monmap changed)... 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: Reconfiguring daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2569426703' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:13.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2049455059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: Reconfiguring rgw.foo.smithi104.iurqzi (monmap changed)... 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: Reconfiguring daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:13 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2569426703' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:14.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 podman[60900]: 2024-04-06 02:25:13.938367351 +0000 UTC m=+0.598601996 container remove 2d03a526484ea4b338d9c0dbf9e2d208b7610e22a2b7ff0e7341a2da32fb35fa (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, version=8.5, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762) 2024-04-06T02:25:14.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:25:14.161 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:13 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2049455059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: Reconfiguring rgw.foo.smithi104.iurqzi (monmap changed)... 2024-04-06T02:25:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: Reconfiguring daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:25:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='client.54163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:14.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:13 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2569426703' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:15.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:14 smithi104 ceph-mon[58668]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:14 smithi017 ceph-mon[64459]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:14 smithi017 ceph-mon[67002]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: Reconfiguring rgw.smpl.smithi104.smkgdi (monmap changed)... 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: Reconfiguring daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:25:16.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: Reconfiguring rgw.smpl.smithi104.smkgdi (monmap changed)... 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: Reconfiguring daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: Reconfiguring rgw.smpl.smithi104.smkgdi (monmap changed)... 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: Reconfiguring daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:25:16.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:17.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:17.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-06T02:25:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-06T02:25:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:17.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:17.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all mon 2024-04-06T02:25:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all crash 2024-04-06T02:25:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[64459]: Upgrade: osd.0 is safe to restart 2024-04-06T02:25:18.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all mon 2024-04-06T02:25:18.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all crash 2024-04-06T02:25:18.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:18.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-mon[67002]: Upgrade: osd.0 is safe to restart 2024-04-06T02:25:18.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:18 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all mon 2024-04-06T02:25:18.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:18 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all crash 2024-04-06T02:25:18.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:18 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-06T02:25:18.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:18 smithi104 ceph-mon[58668]: Upgrade: osd.0 is safe to restart 2024-04-06T02:25:18.895 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:18] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-04-06T02:25:18.905 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:19.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[67002]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:19.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:19.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:25:19.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:19.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[64459]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:19.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:19.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:25:19.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:19.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:19 smithi104 ceph-mon[58668]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:19.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:19 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:19.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:19 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:25:19.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:19 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:20.295 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:20 smithi017 ceph-mon[64459]: Upgrade: Updating osd.0 2024-04-06T02:25:20.295 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:20 smithi017 ceph-mon[64459]: Deploying daemon osd.0 on smithi017 2024-04-06T02:25:20.295 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:20 smithi017 ceph-mon[67002]: Upgrade: Updating osd.0 2024-04-06T02:25:20.295 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:20 smithi017 ceph-mon[67002]: Deploying daemon osd.0 on smithi017 2024-04-06T02:25:20.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:20 smithi104 ceph-mon[58668]: Upgrade: Updating osd.0 2024-04-06T02:25:20.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:20 smithi104 ceph-mon[58668]: Deploying daemon osd.0 on smithi017 2024-04-06T02:25:21.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:21 smithi017 ceph-mon[64459]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:21.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:21 smithi017 ceph-mon[67002]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:21.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:21 smithi104 ceph-mon[58668]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:21.733 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:21 smithi017 systemd[1]: Stopping Ceph osd.0 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:21.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:21 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[32424]: 2024-04-06T02:25:21.795+0000 7feb1699e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:25:21.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:21 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[32424]: 2024-04-06T02:25:21.795+0000 7feb1699e700 -1 osd.0 105 *** Got signal Terminated *** 2024-04-06T02:25:21.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:21 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[32424]: 2024-04-06T02:25:21.795+0000 7feb1699e700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:25:22.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:22 smithi017 ceph-mon[67002]: osd.0 marked itself down and dead 2024-04-06T02:25:22.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:22 smithi017 ceph-mon[64459]: osd.0 marked itself down and dead 2024-04-06T02:25:22.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:22 smithi104 ceph-mon[58668]: osd.0 marked itself down and dead 2024-04-06T02:25:22.856 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:22 smithi017 podman[69899]: 2024-04-06 02:25:22.601048979 +0000 UTC m=+0.935047410 container died a0d080c7a74cb44c644bd849ba3c94507ff3976e0fb1752a0db769dae1cea808 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, distribution-scope=public, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_BRANCH=HEAD, architecture=x86_64, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870) 2024-04-06T02:25:23.184 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:22 smithi017 podman[69899]: 2024-04-06 02:25:22.970720098 +0000 UTC m=+1.304718520 container cleanup a0d080c7a74cb44c644bd849ba3c94507ff3976e0fb1752a0db769dae1cea808 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-04-06T02:25:23.185 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:22 smithi017 bash[69899]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T02:25:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[67002]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[67002]: osdmap e106: 8 total, 7 up, 8 in 2024-04-06T02:25:23.487 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[69927]: 2024-04-06 02:25:23.230354789 +0000 UTC m=+0.628415139 container remove a0d080c7a74cb44c644bd849ba3c94507ff3976e0fb1752a0db769dae1cea808 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.component=centos-stream-container, release=754, vcs-type=git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-06T02:25:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[64459]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:23 smithi017 ceph-mon[64459]: osdmap e106: 8 total, 7 up, 8 in 2024-04-06T02:25:23.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:23 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:23 smithi104 ceph-mon[58668]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:23 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:23.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:23 smithi104 ceph-mon[58668]: osdmap e106: 8 total, 7 up, 8 in 2024-04-06T02:25:23.926 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[70023]: 2024-04-06 02:25:23.485107225 +0000 UTC m=+0.018517434 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:23.926 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[70023]: 2024-04-06 02:25:23.606473744 +0000 UTC m=+0.139883943 container create 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235) 2024-04-06T02:25:23.926 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[70023]: 2024-04-06 02:25:23.817371604 +0000 UTC m=+0.350781804 container init 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:25:23.926 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[70023]: 2024-04-06 02:25:23.820497072 +0000 UTC m=+0.353907275 container start 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235) 2024-04-06T02:25:24.192 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:23 smithi017 podman[70023]: 2024-04-06 02:25:23.942645003 +0000 UTC m=+0.476055215 container attach 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-06T02:25:24.192 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 podman[70023]: 2024-04-06 02:25:24.12018168 +0000 UTC m=+0.653591886 container died 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-06T02:25:24.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:24 smithi017 ceph-mon[64459]: osdmap e107: 8 total, 7 up, 8 in 2024-04-06T02:25:24.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:24 smithi017 ceph-mon[67002]: osdmap e107: 8 total, 7 up, 8 in 2024-04-06T02:25:24.504 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:24 smithi104 ceph-mon[58668]: osdmap e107: 8 total, 7 up, 8 in 2024-04-06T02:25:24.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 13. 2024-04-06T02:25:24.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:24.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 systemd[1]: Starting Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:24.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 podman[61160]: 2024-04-06 02:25:24.31797935 +0000 UTC m=+0.017464266 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-06T02:25:24.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 podman[61160]: 2024-04-06 02:25:24.420945054 +0000 UTC m=+0.120429977 container create 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, vcs-type=git, release=236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., version=8.5, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 podman[61160]: 2024-04-06 02:25:24.681713227 +0000 UTC m=+0.381198143 container init 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, name=ubi8, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container) 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 podman[61160]: 2024-04-06 02:25:24.684515565 +0000 UTC m=+0.384000489 container start 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, maintainer=Paul Cuzner , name=ubi8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.openshift.expose-services=) 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[61175]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a[61175]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 bash[61160]: 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 2024-04-06T02:25:24.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 systemd[1]: Started Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:24.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 podman[70049]: 2024-04-06 02:25:24.746480948 +0000 UTC m=+0.616336664 container remove 2fdaf7bb28656145725f0d06cdce67ef19f2063cbca7a9a1116afab9c839617f (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-06T02:25:24.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-04-06T02:25:24.987 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 systemd[1]: Stopped Ceph osd.0 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:24.988 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.0.service: Consumed 7.489s CPU time. 2024-04-06T02:25:24.988 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:24 smithi017 systemd[1]: Starting Ceph osd.0 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:25.051 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:24 smithi104 podman[61191]: 2024-04-06 02:25:24.773598408 +0000 UTC m=+0.051838675 container died 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, distribution-scope=public, name=ubi8, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, description=Ceph Grafana Container, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-06T02:25:25.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:25 smithi104 ceph-mon[58668]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:25.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-mon[64459]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:25.433 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-mon[67002]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:25.433 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.094975439 +0000 UTC m=+0.017643095 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:25.434 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.197273463 +0000 UTC m=+0.119941111 container create 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-06T02:25:25.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:25 smithi104 podman[61191]: 2024-04-06 02:25:25.360312942 +0000 UTC m=+0.638553209 container remove 87ea876f945e97eae8198c2471d13d3312ac891eb3d19b95dbde7f8f962aa174 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, architecture=x86_64, distribution-scope=public, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, description=Ceph Grafana Container, io.openshift.expose-services=, io.openshift.tags=base rhel8) 2024-04-06T02:25:25.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:25 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-06T02:25:25.661 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:25 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:25:25.737 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.450366575 +0000 UTC m=+0.373034243 container init 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:25:25.737 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.453896841 +0000 UTC m=+0.376564498 container start 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-06T02:25:25.737 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.550010274 +0000 UTC m=+0.472677938 container attach 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-06T02:25:26.186 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T02:25:26.186 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T02:25:26.186 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T02:25:26.186 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T02:25:26.186 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 bash[70149]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate[70163]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-06T02:25:26.187 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:25 smithi017 podman[70149]: 2024-04-06 02:25:25.919834707 +0000 UTC m=+0.842502356 container died 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-06T02:25:26.823 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 podman[70285]: 2024-04-06 02:25:26.521090509 +0000 UTC m=+0.596612072 container remove 6edb0eb2db1ad1e4b056329ff22302de7d71237b004dcce9ce520cf99f1837ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-06T02:25:26.824 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 podman[70344]: 2024-04-06 02:25:26.637469352 +0000 UTC m=+0.018499560 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:26.824 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 podman[70344]: 2024-04-06 02:25:26.738629967 +0000 UTC m=+0.119660204 container create d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-9f19235, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-06T02:25:27.196 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 podman[70344]: 2024-04-06 02:25:26.933744649 +0000 UTC m=+0.314774845 container init d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-06T02:25:27.196 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 podman[70344]: 2024-04-06 02:25:26.936763849 +0000 UTC m=+0.317794046 container start d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:25:27.196 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:26 smithi017 ceph-osd[70363]: -- 172.21.15.17:0/3332903779 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dea95ef680 con 0x55dea95b8c00 2024-04-06T02:25:27.196 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:27 smithi017 bash[70344]: d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c 2024-04-06T02:25:27.196 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:27 smithi017 systemd[1]: Started Ceph osd.0 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:27.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[67002]: pgmap v51: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:27.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[64459]: pgmap v51: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:27 smithi104 ceph-mon[58668]: pgmap v51: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:27.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:27.737 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:27 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:25:27.479+0000 7fb0b3eba7c0 -1 Falling back to public interface 2024-04-06T02:25:28.626 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:28 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 74/723 objects degraded (10.235%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:28.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:28 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 74/723 objects degraded (10.235%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:28.642 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:28 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 74/723 objects degraded (10.235%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:28.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:28.914 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:28] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-04-06T02:25:29.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:29 smithi017 ceph-mon[67002]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:29.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:29 smithi017 ceph-mon[64459]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:29.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:29 smithi104 ceph-mon[58668]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:30.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:30 smithi017 ceph-mon[64459]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:25:30.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:30 smithi017 ceph-mon[67002]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:25:30.488 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:25:30.208+0000 7fb0b3eba7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-06T02:25:30.488 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:25:30.233+0000 7fb0b3eba7c0 -1 osd.0 105 log_to_monitors true 2024-04-06T02:25:30.488 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:25:30 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:25:30.321+0000 7fb0a5312700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:25:30.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:30 smithi104 ceph-mon[58668]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: pgmap v53: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: osdmap e108: 8 total, 7 up, 8 in 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: pgmap v53: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: osdmap e108: 8 total, 7 up, 8 in 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:31.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: pgmap v53: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: osdmap e108: 8 total, 7 up, 8 in 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: from='osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017] boot 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: osdmap e109: 8 total, 8 up, 8 in 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: pgmap v56: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:32 smithi104 ceph-mon[58668]: osdmap e110: 8 total, 8 up, 8 in 2024-04-06T02:25:32.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:32.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017] boot 2024-04-06T02:25:32.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: osdmap e109: 8 total, 8 up, 8 in 2024-04-06T02:25:32.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:25:32.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: pgmap v56: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[67002]: osdmap e110: 8 total, 8 up, 8 in 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: osd.0 [v2:172.21.15.17:6802/363007017,v1:172.21.15.17:6803/363007017] boot 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: osdmap e109: 8 total, 8 up, 8 in 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: pgmap v56: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-04-06T02:25:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:32 smithi017 ceph-mon[64459]: osdmap e110: 8 total, 8 up, 8 in 2024-04-06T02:25:33.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:33.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:33.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:33.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:34.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:34.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:35.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:34 smithi104 ceph-mon[58668]: pgmap v58: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-04-06T02:25:35.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:34 smithi017 ceph-mon[64459]: pgmap v58: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-04-06T02:25:35.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:34 smithi017 ceph-mon[67002]: pgmap v58: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-04-06T02:25:35.830 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:35 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Scheduled restart job, restart counter is at 14. 2024-04-06T02:25:35.831 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:35 smithi104 systemd[1]: Stopped Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:35.831 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:35 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Start request repeated too quickly. 2024-04-06T02:25:35.831 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:35 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@grafana.a.service: Failed with result 'exit-code'. 2024-04-06T02:25:35.831 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Apr 06 02:25:35 smithi104 systemd[1]: Failed to start Ceph grafana.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:35.908 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:35 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-04-06T02:25:35.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:35 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-04-06T02:25:36.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:35 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: pgmap v59: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: pgmap v59: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: pgmap v59: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:37.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:38.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:37 smithi104 ceph-mon[58668]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:25:38.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:37 smithi104 ceph-mon[58668]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-06T02:25:38.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:37 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:38.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:37 smithi104 ceph-mon[58668]: Upgrade: osd.1 is safe to restart 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[67002]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[67002]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[67002]: Upgrade: osd.1 is safe to restart 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[64459]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[64459]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-06T02:25:38.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:37 smithi017 ceph-mon[64459]: Upgrade: osd.1 is safe to restart 2024-04-06T02:25:38.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:38.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:25:38.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:38.916 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:38] "GET /metrics HTTP/1.1" 200 38255 "" "Prometheus/2.33.4" 2024-04-06T02:25:39.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: Upgrade: Updating osd.1 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: Deploying daemon osd.1 on smithi017 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[64459]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: Upgrade: Updating osd.1 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: Deploying daemon osd.1 on smithi017 2024-04-06T02:25:39.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:39 smithi017 ceph-mon[67002]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: Upgrade: Updating osd.1 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: Deploying daemon osd.1 on smithi017 2024-04-06T02:25:39.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:39 smithi104 ceph-mon[58668]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-06T02:25:41.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:41 smithi017 ceph-mon[67002]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:25:41.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:41 smithi017 ceph-mon[64459]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:25:41.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:41 smithi104 ceph-mon[58668]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-06T02:25:42.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:41 smithi017 systemd[1]: Stopping Ceph osd.1 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:42.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[35392]: 2024-04-06T02:25:42.088+0000 7ff0b9490700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:25:42.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[35392]: 2024-04-06T02:25:42.088+0000 7ff0b9490700 -1 osd.1 110 *** Got signal Terminated *** 2024-04-06T02:25:42.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[35392]: 2024-04-06T02:25:42.088+0000 7ff0b9490700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:25:42.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:42 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:42.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:42 smithi104 ceph-mon[58668]: osd.1 marked itself down and dead 2024-04-06T02:25:42.919 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:42.919 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-mon[67002]: osd.1 marked itself down and dead 2024-04-06T02:25:42.919 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:42.919 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:42 smithi017 ceph-mon[64459]: osd.1 marked itself down and dead 2024-04-06T02:25:43.215 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:42 smithi017 podman[72205]: 2024-04-06 02:25:42.936026621 +0000 UTC m=+0.980986816 container died 4385f8547400a5efafb03b9be3f47505355cdc640e893b41bcc7bf44187dc31c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, version=8, GIT_BRANCH=HEAD, ceph=True, release=754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-type=git) 2024-04-06T02:25:43.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:43 smithi017 podman[72205]: 2024-04-06 02:25:43.313798849 +0000 UTC m=+1.358759048 container cleanup 4385f8547400a5efafb03b9be3f47505355cdc640e893b41bcc7bf44187dc31c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, version=8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, com.redhat.component=centos-stream-container, ceph=True) 2024-04-06T02:25:43.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:43 smithi017 bash[72205]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T02:25:43.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[64459]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:25:43.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:43.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[64459]: osdmap e111: 8 total, 7 up, 8 in 2024-04-06T02:25:43.898 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[67002]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:25:43.899 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:43.899 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:43 smithi017 ceph-mon[67002]: osdmap e111: 8 total, 7 up, 8 in 2024-04-06T02:25:43.899 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:43 smithi017 podman[72223]: 2024-04-06 02:25:43.556760687 +0000 UTC m=+0.618621667 container remove 4385f8547400a5efafb03b9be3f47505355cdc640e893b41bcc7bf44187dc31c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, release=754, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8) 2024-04-06T02:25:43.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:43 smithi104 ceph-mon[58668]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:25:43.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:43 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:25:43.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:43 smithi104 ceph-mon[58668]: osdmap e111: 8 total, 7 up, 8 in 2024-04-06T02:25:44.047 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:25:44.235 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:43 smithi017 podman[72318]: 2024-04-06 02:25:43.818865852 +0000 UTC m=+0.020018355 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:44.235 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:43 smithi017 podman[72318]: 2024-04-06 02:25:43.953143369 +0000 UTC m=+0.154295872 container create 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-06T02:25:44.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:44 smithi017 podman[72318]: 2024-04-06 02:25:44.252473045 +0000 UTC m=+0.453625542 container init 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-06T02:25:44.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:44 smithi017 podman[72318]: 2024-04-06 02:25:44.257469113 +0000 UTC m=+0.458621608 container start 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0) 2024-04-06T02:25:44.487 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:44 smithi017 podman[72318]: 2024-04-06 02:25:44.361032436 +0000 UTC m=+0.562184935 container attach 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:25:44.727 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:25:44.727 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 13s ago 8m 24.4M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 61s ago 7m - - 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (2m) 13s ago 7m 52.6M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (2m) 61s ago 12m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (2m) 13s ago 14m 502M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (113s) 13s ago 14m 48.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (71s) 61s ago 12m 16.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (88s) 13s ago 13m 36.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 13s ago 8m 19.6M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (3m) 61s ago 8m 17.7M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (18s) 13s ago 12m 14.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (12m) 13s ago 12m 56.9M 2737M 17.2.0 e1d6a67b021e 4385f8547400 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (11m) 13s ago 11m 50.8M 2737M 17.2.0 e1d6a67b021e 3d3e8c0ec83e 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (11m) 13s ago 11m 57.4M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:25:44.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (10m) 61s ago 10m 55.7M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (10m) 61s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (10m) 61s ago 10m 53.0M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (9m) 61s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (2m) 61s ago 9m 65.0M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (7m) 13s ago 7m 73.7M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (7m) 61s ago 7m 73.0M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (7m) 13s ago 7m 76.3M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:25:44.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (7m) 61s ago 7m 77.3M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:25:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:44 smithi104 ceph-mon[58668]: osdmap e112: 8 total, 7 up, 8 in 2024-04-06T02:25:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:44 smithi104 ceph-mon[58668]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:44.972 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:44 smithi017 ceph-mon[67002]: osdmap e112: 8 total, 7 up, 8 in 2024-04-06T02:25:44.972 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:44 smithi017 ceph-mon[67002]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:44.973 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:44 smithi017 podman[72318]: 2024-04-06 02:25:44.55916097 +0000 UTC m=+0.760313472 container died 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-06T02:25:44.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:44 smithi017 ceph-mon[64459]: osdmap e112: 8 total, 7 up, 8 in 2024-04-06T02:25:44.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:44 smithi017 ceph-mon[64459]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.138 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 1 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:25:45.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-06T02:25:45.140 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 6 2024-04-06T02:25:45.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:25:45.140 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:25:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "6/23 daemons upgraded", 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:25:45.494 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:25:45.496 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 podman[72411]: 2024-04-06 02:25:45.262580768 +0000 UTC m=+0.698266567 container remove 38c3642d0b351ff44e0c265cb88265698468ff8e3c386c599b2eef783e8b7e0c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-9f19235, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T02:25:45.496 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-04-06T02:25:45.496 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 systemd[1]: Stopped Ceph osd.1 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:45.496 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.1.service: Consumed 8.638s CPU time. 2024-04-06T02:25:45.496 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 systemd[1]: Starting Ceph osd.1 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:25:45.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:45.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1199341793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:45.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1199341793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.854 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:45.855 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.855 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 podman[72564]: 2024-04-06 02:25:45.636539841 +0000 UTC m=+0.020611628 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:45.855 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:45 smithi017 podman[72564]: 2024-04-06 02:25:45.763751925 +0000 UTC m=+0.147823708 container create 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1199341793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:25:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:45.918 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down 2024-04-06T02:25:45.918 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:25:45.918 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:25:45.918 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T02:25:45.918 INFO:teuthology.orchestra.run.smithi017.stdout: osd.1 (root=default,host=smithi017) is down 2024-04-06T02:25:46.107 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 podman[72564]: 2024-04-06 02:25:46.033122823 +0000 UTC m=+0.417194599 container init 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-06T02:25:46.107 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 podman[72564]: 2024-04-06 02:25:46.036282813 +0000 UTC m=+0.420354589 container start 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-06T02:25:46.107 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 podman[72564]: 2024-04-06 02:25:46.124620144 +0000 UTC m=+0.508691929 container attach 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, ceph=True, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:25:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-mon[67002]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2095064187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:46.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-mon[64459]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:46.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2095064187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T02:25:46.738 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 bash[72564]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate[72598]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-06T02:25:46.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:46 smithi017 podman[72564]: 2024-04-06 02:25:46.506828745 +0000 UTC m=+0.890900519 container died 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:25:46.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:46 smithi104 ceph-mon[58668]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:25:46.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:46 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2095064187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:25:47.469 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 podman[72733]: 2024-04-06 02:25:47.153801142 +0000 UTC m=+0.637599726 container remove 32b1bee2ba32193b09b203b246b1699b978ad1ec773b84c0b83d636222e7b614 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-06T02:25:47.469 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 podman[72798]: 2024-04-06 02:25:47.265378592 +0000 UTC m=+0.017963005 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:25:47.469 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 podman[72798]: 2024-04-06 02:25:47.379779726 +0000 UTC m=+0.132364125 container create b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:25:47.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:47 smithi104 ceph-mon[58668]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:47.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:47 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:47.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:47 smithi017 ceph-mon[67002]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:47.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:47 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:47.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:47 smithi017 ceph-mon[64459]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:47.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:47 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:47.988 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 podman[72798]: 2024-04-06 02:25:47.623859228 +0000 UTC m=+0.376443628 container init b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:25:47.988 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 podman[72798]: 2024-04-06 02:25:47.627061908 +0000 UTC m=+0.379646307 container start b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T02:25:47.988 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 ceph-osd[72817]: -- 172.21.15.17:0/3653421214 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561834f39680 con 0x561834f03400 2024-04-06T02:25:47.988 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 bash[72798]: b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be 2024-04-06T02:25:47.988 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:47 smithi017 systemd[1]: Started Ceph osd.1 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:25:48.488 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:25:48.162+0000 7fb2685a77c0 -1 Falling back to public interface 2024-04-06T02:25:48.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:25:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:48.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:48] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.33.4" 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:48.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:49.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:49 smithi017 ceph-mon[67002]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:49.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:49 smithi017 ceph-mon[64459]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:50.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:49 smithi104 ceph-mon[58668]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:51.237 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:50 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:25:50.892+0000 7fb2685a77c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-06T02:25:51.237 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:50 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:25:50.925+0000 7fb2685a77c0 -1 osd.1 110 log_to_monitors true 2024-04-06T02:25:52.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:51 smithi104 ceph-mon[58668]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:52.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:51 smithi104 ceph-mon[58668]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:25:52.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:51 smithi017 ceph-mon[67002]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:52.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:51 smithi017 ceph-mon[67002]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:25:52.238 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:25:51 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:25:51.802+0000 7fb2599ff700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:25:52.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:51 smithi017 ceph-mon[64459]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:52.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:51 smithi017 ceph-mon[64459]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:25:53.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:53.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:25:53.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: osdmap e113: 8 total, 7 up, 8 in 2024-04-06T02:25:53.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:53.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:53.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: osdmap e113: 8 total, 7 up, 8 in 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:25:53.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:25:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: osdmap e113: 8 total, 7 up, 8 in 2024-04-06T02:25:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: from='osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:25:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:54.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:54.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320] boot 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: osdmap e114: 8 total, 8 up, 8 in 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320] boot 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: osdmap e114: 8 total, 8 up, 8 in 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-04-06T02:25:54.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:25:54.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: osd.1 [v2:172.21.15.17:6810/3116805320,v1:172.21.15.17:6811/3116805320] boot 2024-04-06T02:25:54.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: osdmap e114: 8 total, 8 up, 8 in 2024-04-06T02:25:54.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:25:54.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:54.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:55.130 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:54 smithi017 ceph-mon[67002]: osdmap e115: 8 total, 8 up, 8 in 2024-04-06T02:25:55.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:54 smithi017 ceph-mon[64459]: osdmap e115: 8 total, 8 up, 8 in 2024-04-06T02:25:55.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:54 smithi104 ceph-mon[58668]: osdmap e115: 8 total, 8 up, 8 in 2024-04-06T02:25:56.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:55 smithi017 ceph-mon[64459]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 100/723 objects degraded (13.831%) 2024-04-06T02:25:56.126 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:55 smithi017 ceph-mon[67002]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 100/723 objects degraded (13.831%) 2024-04-06T02:25:56.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:55 smithi104 ceph-mon[58668]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 100/723 objects degraded (13.831%) 2024-04-06T02:25:57.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/723 objects degraded (5.256%) 2024-04-06T02:25:57.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:57.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[64459]: Upgrade: osd.2 is safe to restart 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/723 objects degraded (5.256%) 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:57.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:57.990 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:57 smithi017 ceph-mon[67002]: Upgrade: osd.2 is safe to restart 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: pgmap v76: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/723 objects degraded (5.256%) 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded (PG_DEGRADED) 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:58.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:25:58.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:58.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-06T02:25:58.162 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:57 smithi104 ceph-mon[58668]: Upgrade: osd.2 is safe to restart 2024-04-06T02:25:58.783 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:25:58.783 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:25:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:25:58.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:25:58] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.33.4" 2024-04-06T02:25:58.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded) 2024-04-06T02:25:58.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:58.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:25:58.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded) 2024-04-06T02:25:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:25:58.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:59.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/723 objects degraded (5.256%), 7 pgs degraded) 2024-04-06T02:25:59.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:25:59.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:25:59.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:25:59.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[67002]: Upgrade: Updating osd.2 2024-04-06T02:25:59.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[67002]: Deploying daemon osd.2 on smithi017 2024-04-06T02:25:59.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[67002]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-04-06T02:25:59.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[64459]: Upgrade: Updating osd.2 2024-04-06T02:25:59.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[64459]: Deploying daemon osd.2 on smithi017 2024-04-06T02:25:59.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:25:59 smithi017 ceph-mon[64459]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-04-06T02:26:00.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:59 smithi104 ceph-mon[58668]: Upgrade: Updating osd.2 2024-04-06T02:26:00.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:59 smithi104 ceph-mon[58668]: Deploying daemon osd.2 on smithi017 2024-04-06T02:26:00.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:25:59 smithi104 ceph-mon[58668]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-04-06T02:26:01.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:01.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:01.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:01 smithi104 ceph-mon[58668]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[64459]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:01.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:01 smithi017 ceph-mon[67002]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-04-06T02:26:02.044 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:01 smithi017 systemd[1]: Stopping Ceph osd.2 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:26:02.400 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[38326]: 2024-04-06T02:26:02.060+0000 7fdfb8ee3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:26:02.400 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[38326]: 2024-04-06T02:26:02.060+0000 7fdfb8ee3700 -1 osd.2 115 *** Got signal Terminated *** 2024-04-06T02:26:02.400 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[38326]: 2024-04-06T02:26:02.060+0000 7fdfb8ee3700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:26:02.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:02 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:02.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:02 smithi104 ceph-mon[58668]: osd.2 marked itself down and dead 2024-04-06T02:26:02.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:02.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-mon[67002]: osd.2 marked itself down and dead 2024-04-06T02:26:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:02 smithi017 ceph-mon[64459]: osd.2 marked itself down and dead 2024-04-06T02:26:03.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:02 smithi017 podman[74672]: 2024-04-06 02:26:02.949912699 +0000 UTC m=+0.993817365 container died 3d3e8c0ec83eec1555760ca27c179093944ecc5b19e6b2f73bb27fd8b5c08496 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, distribution-scope=public, io.buildah.version=1.19.8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream) 2024-04-06T02:26:03.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:03.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[67002]: osdmap e116: 8 total, 7 up, 8 in 2024-04-06T02:26:03.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[67002]: pgmap v82: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-04-06T02:26:03.545 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:03 smithi017 podman[74672]: 2024-04-06 02:26:03.377644585 +0000 UTC m=+1.421549251 container cleanup 3d3e8c0ec83eec1555760ca27c179093944ecc5b19e6b2f73bb27fd8b5c08496 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , release=754) 2024-04-06T02:26:03.545 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:03 smithi017 bash[74672]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T02:26:03.546 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:03.546 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[64459]: osdmap e116: 8 total, 7 up, 8 in 2024-04-06T02:26:03.546 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:03 smithi017 ceph-mon[64459]: pgmap v82: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-04-06T02:26:03.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:03 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:03.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:03 smithi104 ceph-mon[58668]: osdmap e116: 8 total, 7 up, 8 in 2024-04-06T02:26:03.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:03 smithi104 ceph-mon[58668]: pgmap v82: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-04-06T02:26:03.977 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:03 smithi017 podman[74689]: 2024-04-06 02:26:03.562409886 +0000 UTC m=+0.603893713 container remove 3d3e8c0ec83eec1555760ca27c179093944ecc5b19e6b2f73bb27fd8b5c08496 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, vcs-type=git, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, name=centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, ceph=True, distribution-scope=public, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T02:26:04.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:03 smithi017 podman[74779]: 2024-04-06 02:26:03.898319006 +0000 UTC m=+0.018553034 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:04.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:04 smithi017 podman[74779]: 2024-04-06 02:26:04.030373079 +0000 UTC m=+0.150607107 container create dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:26:04.547 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:04 smithi017 ceph-mon[64459]: osdmap e117: 8 total, 7 up, 8 in 2024-04-06T02:26:04.547 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:04 smithi017 ceph-mon[67002]: osdmap e117: 8 total, 7 up, 8 in 2024-04-06T02:26:04.548 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:04 smithi017 podman[74779]: 2024-04-06 02:26:04.257726627 +0000 UTC m=+0.377960657 container init dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-06T02:26:04.548 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:04 smithi017 podman[74779]: 2024-04-06 02:26:04.261282702 +0000 UTC m=+0.381516736 container start dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-06T02:26:04.548 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:04 smithi017 podman[74779]: 2024-04-06 02:26:04.341198785 +0000 UTC m=+0.461432826 container attach dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.license=GPLv2) 2024-04-06T02:26:04.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:04 smithi104 ceph-mon[58668]: osdmap e117: 8 total, 7 up, 8 in 2024-04-06T02:26:04.902 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:04 smithi017 podman[74779]: 2024-04-06 02:26:04.564800421 +0000 UTC m=+0.685034457 container died dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-06T02:26:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:05 smithi017 ceph-mon[67002]: pgmap v84: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:26:05.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:05 smithi017 ceph-mon[64459]: pgmap v84: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:26:05.708 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 podman[74805]: 2024-04-06 02:26:05.279010271 +0000 UTC m=+0.709015928 container remove dd0d44b23f94613992da565540aa4394b8d14500a788d30151548d54225abd78 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-06T02:26:05.708 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-04-06T02:26:05.708 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 systemd[1]: Stopped Ceph osd.2 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:26:05.708 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.2.service: Consumed 6.413s CPU time. 2024-04-06T02:26:05.708 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 systemd[1]: Starting Ceph osd.2 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:26:05.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:05 smithi104 ceph-mon[58668]: pgmap v84: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:26:05.987 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 podman[74911]: 2024-04-06 02:26:05.62797486 +0000 UTC m=+0.018493834 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:05.987 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 podman[74911]: 2024-04-06 02:26:05.744930316 +0000 UTC m=+0.135449309 container create 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-06T02:26:05.987 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 podman[74911]: 2024-04-06 02:26:05.970064115 +0000 UTC m=+0.360583072 container init 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:26:05.987 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:05 smithi017 podman[74911]: 2024-04-06 02:26:05.973438962 +0000 UTC m=+0.363957925 container start 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-06T02:26:06.401 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 podman[74911]: 2024-04-06 02:26:06.045234878 +0000 UTC m=+0.435753845 container attach 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-06T02:26:06.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:06.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 bash[74911]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T02:26:06.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate[74925]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-06T02:26:06.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:06 smithi017 podman[74911]: 2024-04-06 02:26:06.442095313 +0000 UTC m=+0.832614272 container died 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T02:26:06.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 podman[75056]: 2024-04-06 02:26:07.17417638 +0000 UTC m=+0.727205356 container remove 9e63563d22b957ab1d8e885c2b5f58b94ee9d7a37909536198e83008d29bc8ec (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-06T02:26:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 podman[75121]: 2024-04-06 02:26:07.391813875 +0000 UTC m=+0.113656640 container create 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-06T02:26:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 podman[75121]: 2024-04-06 02:26:07.296273751 +0000 UTC m=+0.018116520 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:07 smithi017 ceph-mon[67002]: pgmap v85: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:07 smithi017 ceph-mon[64459]: pgmap v85: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:07.737 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 podman[75121]: 2024-04-06 02:26:07.627789606 +0000 UTC m=+0.349632372 container init 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235) 2024-04-06T02:26:07.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 podman[75121]: 2024-04-06 02:26:07.631014215 +0000 UTC m=+0.352856982 container start 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_CLEAN=True) 2024-04-06T02:26:07.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 ceph-osd[75139]: -- 172.21.15.17:0/1720913774 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a78538d680 con 0x55a785356c00 2024-04-06T02:26:07.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 bash[75121]: 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 2024-04-06T02:26:07.738 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:07 smithi017 systemd[1]: Started Ceph osd.2 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:26:07.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:07 smithi104 ceph-mon[58668]: pgmap v85: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:08.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:26:08.175+0000 7fb2bc7197c0 -1 Falling back to public interface 2024-04-06T02:26:08.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:08] "GET /metrics HTTP/1.1" 200 38416 "" "Prometheus/2.33.4" 2024-04-06T02:26:08.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 64/723 objects degraded (8.852%), 13 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 64/723 objects degraded (8.852%), 13 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:08 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 64/723 objects degraded (8.852%), 13 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:08.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:08.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:09.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:09 smithi017 ceph-mon[67002]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:09.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:09 smithi017 ceph-mon[64459]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:09.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:09 smithi104 ceph-mon[58668]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:11.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:10 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:26:10.889+0000 7fb2bc7197c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-06T02:26:11.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:10 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:26:10.911+0000 7fb2bc7197c0 -1 osd.2 115 log_to_monitors true 2024-04-06T02:26:12.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:11 smithi104 ceph-mon[58668]: pgmap v87: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:12.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:11 smithi104 ceph-mon[58668]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:26:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:11 smithi017 ceph-mon[67002]: pgmap v87: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:11 smithi017 ceph-mon[67002]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:26:12.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:11 smithi017 ceph-mon[64459]: pgmap v87: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:12.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:11 smithi017 ceph-mon[64459]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:26:12.237 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:26:11 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:26:11.890+0000 7fb2adb71700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:26:13.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:13.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:26:13.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: osdmap e118: 8 total, 7 up, 8 in 2024-04-06T02:26:13.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:13.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:13.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: osdmap e118: 8 total, 7 up, 8 in 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:13.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: osdmap e118: 8 total, 7 up, 8 in 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: from='osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:13.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: pgmap v89: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100] boot 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: osdmap e119: 8 total, 8 up, 8 in 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[67002]: osdmap e120: 8 total, 8 up, 8 in 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: pgmap v89: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100] boot 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: osdmap e119: 8 total, 8 up, 8 in 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:13 smithi017 ceph-mon[64459]: osdmap e120: 8 total, 8 up, 8 in 2024-04-06T02:26:14.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: pgmap v89: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: osd.2 [v2:172.21.15.17:6818/2875611100,v1:172.21.15.17:6819/2875611100] boot 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: osdmap e119: 8 total, 8 up, 8 in 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:14.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:13 smithi104 ceph-mon[58668]: osdmap e120: 8 total, 8 up, 8 in 2024-04-06T02:26:16.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:15 smithi104 ceph-mon[58668]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:16.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:15 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:16.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:15 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[67002]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[64459]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:16.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:15 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:16.283 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:26:16.868 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:26:16.800+0000 7fc9f2743700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-06T02:26:16.969 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 4s ago 8m 24.4M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 93s ago 8m - - 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (2m) 4s ago 7m 52.8M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (2m) 93s ago 13m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (3m) 4s ago 14m 505M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (2m) 4s ago 14m 51.5M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (104s) 93s ago 13m 16.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 4s ago 13m 39.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:26:16.970 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 4s ago 9m 19.0M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (3m) 93s ago 9m 17.7M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (50s) 4s ago 12m 46.6M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (29s) 4s ago 12m 42.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (9s) 4s ago 12m 12.6M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (11m) 4s ago 11m 59.4M 2737M 17.2.0 e1d6a67b021e 5abeef6d6cd0 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (11m) 93s ago 11m 55.7M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (11m) 93s ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (10m) 93s ago 10m 53.0M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (10m) 93s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (2m) 93s ago 9m 65.0M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:26:16.971 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (7m) 4s ago 7m 73.9M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:26:16.972 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (7m) 93s ago 7m 73.0M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:26:16.972 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (7m) 4s ago 7m 76.5M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:26:16.972 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (7m) 93s ago 7m 77.3M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:17.375 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:26:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:26:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:26:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:17.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:26:17.376 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-06T02:26:17.377 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T02:26:17.377 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:26:17.377 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:26:17.717 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "8/23 daemons upgraded", 2024-04-06T02:26:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:26:17.718 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:26:17.718 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: pgmap v93: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: pgmap v94: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-06T02:26:17.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1033203181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: pgmap v93: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: pgmap v94: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1033203181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:17 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-04-06T02:26:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-04-06T02:26:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-04-06T02:26:18.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:18.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: pgmap v93: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: pgmap v94: 161 pgs: 3 active+undersized+degraded, 14 active+undersized, 144 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 23/723 objects degraded (3.181%) 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1033203181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:18.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:17 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:18 smithi104 ceph-mon[58668]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:18.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:18 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/414955906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:18.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:18.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:18] "GET /metrics HTTP/1.1" 200 38289 "" "Prometheus/2.33.4" 2024-04-06T02:26:18.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:18 smithi017 ceph-mon[67002]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:18.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:18 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/414955906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:18.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:18 smithi017 ceph-mon[64459]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:18.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:18 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/414955906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:19 smithi104 ceph-mon[58668]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-06T02:26:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:19 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded) 2024-04-06T02:26:20.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:19 smithi017 ceph-mon[67002]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-06T02:26:20.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:19 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded) 2024-04-06T02:26:20.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:19 smithi017 ceph-mon[64459]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-06T02:26:20.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:19 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/723 objects degraded (3.181%), 3 pgs degraded) 2024-04-06T02:26:22.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:21 smithi104 ceph-mon[58668]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-06T02:26:22.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:21 smithi017 ceph-mon[67002]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-06T02:26:22.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:21 smithi017 ceph-mon[64459]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-04-06T02:26:23.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:22 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:23.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:22 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:23.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:22 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:24.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:23 smithi104 ceph-mon[58668]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:26:24.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:23 smithi017 ceph-mon[67002]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:26:24.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:23 smithi017 ceph-mon[64459]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:26:26.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:25 smithi104 ceph-mon[58668]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:26:26.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:25 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:26.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:25 smithi017 ceph-mon[64459]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:26:26.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:25 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:26.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:25 smithi017 ceph-mon[67002]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:26:26.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:25 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:28.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:27 smithi104 ceph-mon[58668]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:28.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:27 smithi017 ceph-mon[64459]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:28.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:27 smithi017 ceph-mon[67002]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:28.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:28.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:28] "GET /metrics HTTP/1.1" 200 38289 "" "Prometheus/2.33.4" 2024-04-06T02:26:30.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:29 smithi104 ceph-mon[58668]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:30.228 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:29 smithi017 ceph-mon[67002]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:30.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:29 smithi017 ceph-mon[64459]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:31.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:31.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:31.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:31 smithi104 ceph-mon[58668]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:31.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:31.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:31.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[67002]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:31.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:31.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:31.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:31 smithi017 ceph-mon[64459]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:32.638 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[67002]: Upgrade: osd.3 is safe to restart 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:32 smithi017 ceph-mon[64459]: Upgrade: osd.3 is safe to restart 2024-04-06T02:26:32.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:32 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:32.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:32 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-06T02:26:32.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:32 smithi104 ceph-mon[58668]: Upgrade: osd.3 is safe to restart 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[64459]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[67002]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:26:33.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:33.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:33 smithi104 ceph-mon[58668]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:33.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:33.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:26:33.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:34.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:34 smithi017 ceph-mon[67002]: Upgrade: Updating osd.3 2024-04-06T02:26:34.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:34 smithi017 ceph-mon[67002]: Deploying daemon osd.3 on smithi017 2024-04-06T02:26:34.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:34 smithi017 ceph-mon[64459]: Upgrade: Updating osd.3 2024-04-06T02:26:34.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:34 smithi017 ceph-mon[64459]: Deploying daemon osd.3 on smithi017 2024-04-06T02:26:34.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:34 smithi104 ceph-mon[58668]: Upgrade: Updating osd.3 2024-04-06T02:26:34.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:34 smithi104 ceph-mon[58668]: Deploying daemon osd.3 on smithi017 2024-04-06T02:26:35.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:35 smithi017 ceph-mon[64459]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:35.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:35 smithi017 ceph-mon[67002]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:35.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:35 smithi104 ceph-mon[58668]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:26:36.929 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:36 smithi017 systemd[1]: Stopping Ceph osd.3 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:26:37.237 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[41309]: 2024-04-06T02:26:36.944+0000 7f3e465be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:26:37.237 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[41309]: 2024-04-06T02:26:36.944+0000 7f3e465be700 -1 osd.3 120 *** Got signal Terminated *** 2024-04-06T02:26:37.237 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[41309]: 2024-04-06T02:26:36.944+0000 7f3e465be700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:26:37.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:37 smithi017 ceph-mon[67002]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:37.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:37 smithi017 ceph-mon[67002]: osd.3 marked itself down and dead 2024-04-06T02:26:37.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:37 smithi017 ceph-mon[64459]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:37.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:37 smithi017 ceph-mon[64459]: osd.3 marked itself down and dead 2024-04-06T02:26:38.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:37 smithi104 ceph-mon[58668]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:26:38.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:37 smithi104 ceph-mon[58668]: osd.3 marked itself down and dead 2024-04-06T02:26:38.487 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:38 smithi017 podman[77141]: 2024-04-06 02:26:38.21036578 +0000 UTC m=+1.485170060 container died 5abeef6d6cd0135a375dba7d9ece1221e7d9f09645b446ff5ff303160cc0a634 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream) 2024-04-06T02:26:38.821 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:38 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:38.821 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:38 smithi017 ceph-mon[67002]: osdmap e121: 8 total, 7 up, 8 in 2024-04-06T02:26:38.822 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:38] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.33.4" 2024-04-06T02:26:38.822 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:38 smithi017 podman[77141]: 2024-04-06 02:26:38.587658647 +0000 UTC m=+1.862462915 container cleanup 5abeef6d6cd0135a375dba7d9ece1221e7d9f09645b446ff5ff303160cc0a634 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-06T02:26:38.822 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:38 smithi017 bash[77141]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T02:26:38.822 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:38 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:38.823 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:38 smithi017 ceph-mon[64459]: osdmap e121: 8 total, 7 up, 8 in 2024-04-06T02:26:38.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:38 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:26:38.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:38 smithi104 ceph-mon[58668]: osdmap e121: 8 total, 7 up, 8 in 2024-04-06T02:26:38.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:39.122 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:38 smithi017 podman[77159]: 2024-04-06 02:26:38.839081214 +0000 UTC m=+0.620684334 container remove 5abeef6d6cd0135a375dba7d9ece1221e7d9f09645b446ff5ff303160cc0a634 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, architecture=x86_64, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., RELEASE=HEAD, version=8) 2024-04-06T02:26:39.400 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.042638746 +0000 UTC m=+0.017668183 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:39.400 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.156629494 +0000 UTC m=+0.131658934 container create 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-06T02:26:39.701 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.418050809 +0000 UTC m=+0.393080251 container init 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:26:39.701 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.422287303 +0000 UTC m=+0.397316739 container start 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-06T02:26:39.701 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.50153441 +0000 UTC m=+0.476563857 container attach 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:26:39.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:39 smithi017 ceph-mon[67002]: pgmap v106: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:39.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:39 smithi017 ceph-mon[67002]: osdmap e122: 8 total, 7 up, 8 in 2024-04-06T02:26:39.987 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:39 smithi017 podman[77251]: 2024-04-06 02:26:39.719172308 +0000 UTC m=+0.694201752 container died 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-06T02:26:39.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:39 smithi017 ceph-mon[64459]: pgmap v106: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:39.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:39 smithi017 ceph-mon[64459]: osdmap e122: 8 total, 7 up, 8 in 2024-04-06T02:26:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:39 smithi104 ceph-mon[58668]: pgmap v106: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:26:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:39 smithi104 ceph-mon[58668]: osdmap e122: 8 total, 7 up, 8 in 2024-04-06T02:26:40.729 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 podman[77276]: 2024-04-06 02:26:40.30532496 +0000 UTC m=+0.577081685 container remove 9444510b333adc8c420c5cc5410470901ba3c05550ee5f6b4499cad00990c27e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-06T02:26:40.729 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-04-06T02:26:40.729 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 systemd[1]: Stopped Ceph osd.3 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:26:40.729 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.3.service: Consumed 8.970s CPU time. 2024-04-06T02:26:40.729 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 systemd[1]: Starting Ceph osd.3 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:26:40.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:40 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:40.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:40 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:40.987 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 podman[77376]: 2024-04-06 02:26:40.648759894 +0000 UTC m=+0.017684683 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:40.987 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:40 smithi017 podman[77376]: 2024-04-06 02:26:40.770909563 +0000 UTC m=+0.139834325 container create 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, ceph=True) 2024-04-06T02:26:41.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:40 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 6 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 podman[77376]: 2024-04-06 02:26:41.021471803 +0000 UTC m=+0.390396573 container init 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235) 2024-04-06T02:26:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 podman[77376]: 2024-04-06 02:26:41.024951269 +0000 UTC m=+0.393876043 container start 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-06T02:26:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 podman[77376]: 2024-04-06 02:26:41.126279871 +0000 UTC m=+0.495204655 container attach 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-06T02:26:41.737 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T02:26:41.737 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T02:26:41.737 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T02:26:41.737 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 bash[77376]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate[77391]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-06T02:26:41.738 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:41 smithi017 podman[77376]: 2024-04-06 02:26:41.499527932 +0000 UTC m=+0.868452699 container died 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T02:26:42.104 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-mon[67002]: pgmap v108: 161 pgs: 6 active+undersized+degraded, 21 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:26:42.104 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:41 smithi017 ceph-mon[64459]: pgmap v108: 161 pgs: 6 active+undersized+degraded, 21 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:26:42.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:41 smithi104 ceph-mon[58668]: pgmap v108: 161 pgs: 6 active+undersized+degraded, 21 stale+active+clean, 8 active+undersized, 126 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:26:42.487 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 podman[77518]: 2024-04-06 02:26:42.122204661 +0000 UTC m=+0.616224323 container remove 197cbd517d29c898050ba48bb22c59a209502fda653054199604b6607cf129c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:26:42.487 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 podman[77583]: 2024-04-06 02:26:42.32291154 +0000 UTC m=+0.105132107 container create 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T02:26:42.487 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 podman[77583]: 2024-04-06 02:26:42.235537353 +0000 UTC m=+0.017757941 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:26:42.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:42 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:42.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:42 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:42.987 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 podman[77583]: 2024-04-06 02:26:42.642101354 +0000 UTC m=+0.424321929 container init 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-06T02:26:42.988 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 podman[77583]: 2024-04-06 02:26:42.645368139 +0000 UTC m=+0.427588708 container start 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-06T02:26:42.988 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 ceph-osd[77602]: -- 172.21.15.17:0/2596513204 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a18dd3680 con 0x563a18d9cc00 2024-04-06T02:26:42.988 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 bash[77583]: 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 2024-04-06T02:26:42.988 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:42 smithi017 systemd[1]: Started Ceph osd.3 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:26:43.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:42 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:43.487 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:26:43.189+0000 7f5dace5e7c0 -1 Falling back to public interface 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[67002]: pgmap v109: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[64459]: pgmap v109: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:43.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:44.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:43 smithi104 ceph-mon[58668]: pgmap v109: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:44.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:44.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:44.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:46.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:45 smithi104 ceph-mon[58668]: pgmap v110: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:46.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:46.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:46.237 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:26:45.890+0000 7f5dace5e7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-06T02:26:46.237 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:26:45.930+0000 7f5dace5e7c0 -1 osd.3 120 log_to_monitors true 2024-04-06T02:26:46.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[67002]: pgmap v110: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:46.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:46.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:46.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[64459]: pgmap v110: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:46.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:46.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:26:47.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:46 smithi104 ceph-mon[58668]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:46 smithi104 ceph-mon[58668]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:46 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:47.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[64459]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[64459]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:47.082 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[67002]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.082 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[67002]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:26:47.082 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:46 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:47.486 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:26:47.098+0000 7f5d9a2ae700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:26:48.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: pgmap v111: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:48.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: osdmap e123: 8 total, 7 up, 8 in 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='osd.3 ' entity='osd.3' 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:47 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: pgmap v111: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: osdmap e123: 8 total, 7 up, 8 in 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='osd.3 ' entity='osd.3' 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: pgmap v111: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: osdmap e123: 8 total, 7 up, 8 in 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='osd.3 ' entity='osd.3' 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:47 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.495 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:26:48.803 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:48] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.33.4" 2024-04-06T02:26:48.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:48.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541] boot 2024-04-06T02:26:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: osdmap e124: 8 total, 8 up, 8 in 2024-04-06T02:26:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:26:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:48.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:26:48.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541] boot 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: osdmap e124: 8 total, 8 up, 8 in 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:49.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:49.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:26:49.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: osd.3 [v2:172.21.15.17:6826/1151104541,v1:172.21.15.17:6827/1151104541] boot 2024-04-06T02:26:49.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: osdmap e124: 8 total, 8 up, 8 in 2024-04-06T02:26:49.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:26:49.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:49.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:48 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 2s ago 9m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 unknown 2m ago 8m - - 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (3m) 2s ago 8m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (3m) 2m ago 13m 449M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:26:49.219 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (3m) 2s ago 15m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (2m) 2s ago 15m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (2m) 2m ago 14m 16.7M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 2s ago 14m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 2s ago 10m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (4m) 2m ago 9m 17.7M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (82s) 2s ago 13m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (61s) 2s ago 13m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (41s) 2s ago 12m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6s) 2s ago 12m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (11m) 2m ago 11m 55.7M 2992M 17.2.0 e1d6a67b021e 8f1e81bdd6bf 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (11m) 2m ago 11m 53.0M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (10m) 2m ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (3m) 2m ago 10m 65.0M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:26:49.220 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (8m) 2s ago 8m 74.0M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:26:49.221 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (8m) 2m ago 8m 73.0M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:26:49.221 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (8m) 2s ago 8m 76.6M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:26:49.221 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (8m) 2m ago 8m 77.3M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:26:49.628 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:26:49.629 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:26:49.630 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-06T02:26:49.630 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 9 2024-04-06T02:26:49.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:26:49.630 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:26:49.977 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:26:49.977 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:26:49.977 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:26:49.977 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:26:49.977 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "9/23 daemons upgraded", 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:26:49.978 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: osdmap e125: 8 total, 8 up, 8 in 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1616488109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: osdmap e125: 8 total, 8 up, 8 in 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:50 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1616488109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-04-06T02:26:50.392 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-04-06T02:26:50.393 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.14 is active+undersized+degraded, acting [5,7] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-04-06T02:26:50.394 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2024-04-06T02:26:50.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-04-06T02:26:50.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: osdmap e125: 8 total, 8 up, 8 in 2024-04-06T02:26:50.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:50.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:50 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1616488109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:51.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:51 smithi017 ceph-mon[67002]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:51.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:51 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3140349088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:51.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:51 smithi017 ceph-mon[64459]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:51.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:51 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3140349088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:51.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:51 smithi104 ceph-mon[58668]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:26:51.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:51 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3140349088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:26:51.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:51 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:26:51.796+0000 7fc9f2743700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T02:26:52.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:52.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:52 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-04-06T02:26:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:26:52.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:52.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:52 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 22 pgs degraded (PG_DEGRADED) 2024-04-06T02:26:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:53 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:53 smithi104 ceph-mon[58668]: pgmap v117: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:53 smithi104 ceph-mon[58668]: pgmap v118: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:53 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:53.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:53 smithi104 ceph-mon[58668]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[67002]: pgmap v117: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[67002]: pgmap v118: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[67002]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T02:26:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:26:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[64459]: pgmap v117: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[64459]: pgmap v118: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 56/723 objects degraded (7.746%) 2024-04-06T02:26:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:26:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:53 smithi017 ceph-mon[64459]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-04-06T02:26:54.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:54 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 15 pgs degraded) 2024-04-06T02:26:54.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:54 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 15 pgs degraded) 2024-04-06T02:26:54.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:54 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 15 pgs degraded) 2024-04-06T02:26:55.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:55 smithi104 ceph-mon[58668]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:55 smithi017 ceph-mon[64459]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:55.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:55 smithi017 ceph-mon[67002]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:57 smithi104 ceph-mon[58668]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:57.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:57 smithi017 ceph-mon[67002]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:57 smithi017 ceph-mon[64459]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:58.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:58.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:26:58.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:26:58 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:26:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:26:58.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:26:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:26:58] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.33.4" 2024-04-06T02:26:59.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:26:59 smithi017 ceph-mon[67002]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:59.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:26:59 smithi017 ceph-mon[64459]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:26:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:26:59 smithi104 ceph-mon[58668]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:01.481 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[67002]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:01.481 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:01.481 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:01.482 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:01.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[64459]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:01.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:01.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:01.482 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:01.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:01 smithi104 ceph-mon[58668]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:01.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:01.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:01.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:03 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:03 smithi017 ceph-mon[67002]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:03.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:03 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:03.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:03 smithi017 ceph-mon[64459]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:03.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:03 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:03.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:03 smithi104 ceph-mon[58668]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:27:05.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:05 smithi017 ceph-mon[64459]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-06T02:27:05.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:05 smithi017 ceph-mon[67002]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-06T02:27:05.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:05 smithi104 ceph-mon[58668]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-06T02:27:07.471 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:07 smithi104 ceph-mon[58668]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:07.471 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:07 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:07.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:07 smithi017 ceph-mon[64459]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:07.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:07.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:07 smithi017 ceph-mon[67002]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:07.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:07 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:08.373 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:08 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:08.373 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:08 smithi104 ceph-mon[58668]: Upgrade: osd.4 is safe to restart 2024-04-06T02:27:08.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:08 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:08.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:08 smithi017 ceph-mon[64459]: Upgrade: osd.4 is safe to restart 2024-04-06T02:27:08.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:08 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-06T02:27:08.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:08 smithi017 ceph-mon[67002]: Upgrade: osd.4 is safe to restart 2024-04-06T02:27:08.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:27:08 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:27:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:27:08.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:08] "GET /metrics HTTP/1.1" 200 38446 "" "Prometheus/2.33.4" 2024-04-06T02:27:09.265 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:09 smithi104 ceph-mon[58668]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:09.266 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:09.266 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:27:09.266 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[67002]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[64459]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:09.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:27:09.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:10.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:10 smithi104 ceph-mon[58668]: Upgrade: Updating osd.4 2024-04-06T02:27:10.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:10 smithi104 ceph-mon[58668]: Deploying daemon osd.4 on smithi104 2024-04-06T02:27:10.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:10 smithi017 ceph-mon[67002]: Upgrade: Updating osd.4 2024-04-06T02:27:10.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:10 smithi017 ceph-mon[67002]: Deploying daemon osd.4 on smithi104 2024-04-06T02:27:10.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:10 smithi017 ceph-mon[64459]: Upgrade: Updating osd.4 2024-04-06T02:27:10.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:10 smithi017 ceph-mon[64459]: Deploying daemon osd.4 on smithi104 2024-04-06T02:27:11.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:11 smithi104 ceph-mon[58668]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:11.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:11 smithi017 ceph-mon[67002]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:11.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:11 smithi017 ceph-mon[64459]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:12.127 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:11 smithi104 systemd[1]: Stopping Ceph osd.4 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:27:12.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:12 smithi104 ceph-mon[58668]: osd.4 marked itself down and dead 2024-04-06T02:27:12.411 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:12 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[29089]: 2024-04-06T02:27:12.165+0000 7f862fa3c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:27:12.411 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:12 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[29089]: 2024-04-06T02:27:12.165+0000 7f862fa3c700 -1 osd.4 125 *** Got signal Terminated *** 2024-04-06T02:27:12.411 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:12 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[29089]: 2024-04-06T02:27:12.165+0000 7f862fa3c700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:27:12.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:12 smithi017 ceph-mon[67002]: osd.4 marked itself down and dead 2024-04-06T02:27:12.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:12 smithi017 ceph-mon[64459]: osd.4 marked itself down and dead 2024-04-06T02:27:13.025 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:12 smithi104 podman[61828]: 2024-04-06 02:27:12.748252735 +0000 UTC m=+0.688482160 container died 8f1e81bdd6bfb4df893cf32cbf02521803f61d0205e52ed0ea5fbdbfc3867716 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, distribution-scope=public, release=754, io.k8s.display-name=CentOS Stream 8, version=8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-06T02:27:13.392 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61828]: 2024-04-06 02:27:13.158210897 +0000 UTC m=+1.098440315 container cleanup 8f1e81bdd6bfb4df893cf32cbf02521803f61d0205e52ed0ea5fbdbfc3867716 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, io.openshift.expose-services=) 2024-04-06T02:27:13.393 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 bash[61828]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T02:27:13.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:13 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:13.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:13 smithi104 ceph-mon[58668]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:13.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:13 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:13.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:13 smithi104 ceph-mon[58668]: osdmap e126: 8 total, 7 up, 8 in 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[67002]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[67002]: osdmap e126: 8 total, 7 up, 8 in 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[64459]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:13.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:13 smithi017 ceph-mon[64459]: osdmap e126: 8 total, 7 up, 8 in 2024-04-06T02:27:13.660 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61855]: 2024-04-06 02:27:13.409757638 +0000 UTC m=+0.656218110 container remove 8f1e81bdd6bfb4df893cf32cbf02521803f61d0205e52ed0ea5fbdbfc3867716 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, version=8, ceph=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc.) 2024-04-06T02:27:13.996 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61949]: 2024-04-06 02:27:13.606045259 +0000 UTC m=+0.018330669 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:13.996 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61949]: 2024-04-06 02:27:13.735672329 +0000 UTC m=+0.147957748 container create fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, ceph=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-06T02:27:13.996 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61949]: 2024-04-06 02:27:13.930829535 +0000 UTC m=+0.343114947 container init fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-06T02:27:13.996 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:13 smithi104 podman[61949]: 2024-04-06 02:27:13.933936606 +0000 UTC m=+0.346222019 container start fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True) 2024-04-06T02:27:13.996 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 podman[61949]: 2024-04-06 02:27:14.013367758 +0000 UTC m=+0.425653168 container attach fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=) 2024-04-06T02:27:14.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:14 smithi104 ceph-mon[58668]: osdmap e127: 8 total, 7 up, 8 in 2024-04-06T02:27:14.410 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 podman[61949]: 2024-04-06 02:27:14.225647632 +0000 UTC m=+0.637933035 container died fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T02:27:14.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:14 smithi017 ceph-mon[67002]: osdmap e127: 8 total, 7 up, 8 in 2024-04-06T02:27:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:14 smithi017 ceph-mon[64459]: osdmap e127: 8 total, 7 up, 8 in 2024-04-06T02:27:15.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 podman[61976]: 2024-04-06 02:27:14.833560306 +0000 UTC m=+0.599656476 container remove fdf622952abb2683275dade82975263cb2bcfe1ec4232a545d570a1433cc5e6d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:27:15.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.4.service: Deactivated successfully. 2024-04-06T02:27:15.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 systemd[1]: Stopped Ceph osd.4 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:27:15.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:14 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.4.service: Consumed 9.419s CPU time. 2024-04-06T02:27:15.161 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 systemd[1]: Starting Ceph osd.4 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:27:15.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:15 smithi017 ceph-mon[67002]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:15.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:15 smithi017 ceph-mon[64459]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-mon[58668]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:15.511 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.183357262 +0000 UTC m=+0.018870479 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:15.511 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.301155267 +0000 UTC m=+0.136668461 container create 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, RELEASE=squid-9f19235, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T02:27:15.511 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.528813715 +0000 UTC m=+0.364326908 container init 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235) 2024-04-06T02:27:15.910 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.532068926 +0000 UTC m=+0.367582130 container start 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-06T02:27:15.910 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.620398121 +0000 UTC m=+0.455911340 container attach 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 bash[62082]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-06T02:27:16.299 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T02:27:16.300 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate[62097]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-06T02:27:16.300 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:15 smithi104 podman[62082]: 2024-04-06 02:27:15.997176215 +0000 UTC m=+0.832689418 container died 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-06T02:27:16.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:16.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:16.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:16 smithi104 ceph-mon[58668]: pgmap v132: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[67002]: pgmap v132: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:16.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:16 smithi017 ceph-mon[64459]: pgmap v132: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 27/723 objects degraded (3.734%) 2024-04-06T02:27:16.905 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:16 smithi104 podman[62214]: 2024-04-06 02:27:16.590967326 +0000 UTC m=+0.583898020 container remove 46c3d366afb5ab1c0fc880431240ddf33767f8b9a35c7e338173ba317c1f2307 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS) 2024-04-06T02:27:16.905 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:16 smithi104 podman[62279]: 2024-04-06 02:27:16.710584686 +0000 UTC m=+0.018258608 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:16.905 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:16 smithi104 podman[62279]: 2024-04-06 02:27:16.816916866 +0000 UTC m=+0.124590825 container create 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=) 2024-04-06T02:27:17.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 podman[62279]: 2024-04-06 02:27:17.044475279 +0000 UTC m=+0.352149196 container init 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-06T02:27:17.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 podman[62279]: 2024-04-06 02:27:17.047458952 +0000 UTC m=+0.355132872 container start 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True) 2024-04-06T02:27:17.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-osd[62299]: -- 172.21.15.104:0/187841024 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d00bd63680 con 0x55d00bd2d400 2024-04-06T02:27:17.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 bash[62279]: 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b 2024-04-06T02:27:17.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 systemd[1]: Started Ceph osd.4 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:27:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:17.910 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:17 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:27:17.594+0000 7f7389a867c0 -1 Falling back to public interface 2024-04-06T02:27:18.642 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:18 smithi104 ceph-mon[58668]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:18.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:18 smithi017 ceph-mon[67002]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:18.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:18 smithi017 ceph-mon[64459]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:18.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:18] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.33.4" 2024-04-06T02:27:18.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:27:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:27:18.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:27:18.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:27:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:27:18.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:27:20.410 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:27:20.297+0000 7f7389a867c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-06T02:27:20.410 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:27:20.337+0000 7f7389a867c0 -1 osd.4 125 log_to_monitors true 2024-04-06T02:27:20.767 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:27:20.866 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-mon[58668]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:20.867 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-mon[58668]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:20.867 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-mon[58668]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:20.867 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:27:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:27:20.771+0000 7f737aede700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:27:20.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[67002]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:20.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[67002]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:20.987 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[67002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:20.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[64459]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:20.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[64459]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:20.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:20 smithi017 ceph-mon[64459]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:27:21.458 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 34s ago 9m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 0s ago 9m - - 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (3m) 34s ago 8m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (3m) 0s ago 14m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (4m) 34s ago 15m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 34s ago 16m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (2m) 0s ago 14m 36.6M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 34s ago 14m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 34s ago 10m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (4m) 0s ago 10m 17.9M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (114s) 34s ago 14m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (94s) 34s ago 13m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:27:21.459 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (74s) 34s ago 13m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (39s) 34s ago 12m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (4s) 0s ago 12m 12.6M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (12m) 0s ago 12m 59.5M 2992M 17.2.0 e1d6a67b021e e704a0a9ebdc 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (11m) 0s ago 11m 57.3M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (11m) 0s ago 11m 59.0M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (3m) 0s ago 10m 69.4M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (9m) 34s ago 9m 74.0M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (9m) 0s ago 9m 73.7M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (8m) 34s ago 8m 76.6M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:27:21.460 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (8m) 0s ago 8m 77.9M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:27:21.884 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:27:21.884 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 10 2024-04-06T02:27:21.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:27:21.886 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:27:22.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:27:22.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: osdmap e128: 8 total, 7 up, 8 in 2024-04-06T02:27:22.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='client.54316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:21 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:27:22.224 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "10/23 daemons upgraded", 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:27:22.225 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:27:22.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:27:22.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: osdmap e128: 8 total, 7 up, 8 in 2024-04-06T02:27:22.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='client.54316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: osdmap e128: 8 total, 7 up, 8 in 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='client.54316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:21 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.637 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded 2024-04-06T02:27:22.637 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:27:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in unknown state 2024-04-06T02:27:22.637 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded 2024-04-06T02:27:22.637 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-04-06T02:27:22.638 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-04-06T02:27:22.639 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='client.54322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952] boot 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: osdmap e129: 8 total, 8 up, 8 in 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2249439897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:22.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='client.54334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3522626043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[64459]: osdmap e130: 8 total, 8 up, 8 in 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='client.54322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952] boot 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: osdmap e129: 8 total, 8 up, 8 in 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2249439897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:22.988 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='client.54334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3522626043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:22.989 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:22 smithi017 ceph-mon[67002]: osdmap e130: 8 total, 8 up, 8 in 2024-04-06T02:27:23.103 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='client.54322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: osd.4 [v2:172.21.15.104:6800/3790988952,v1:172.21.15.104:6801/3790988952] boot 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: osdmap e129: 8 total, 8 up, 8 in 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2249439897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='client.54334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3522626043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:23.104 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:22 smithi104 ceph-mon[58668]: osdmap e130: 8 total, 8 up, 8 in 2024-04-06T02:27:24.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:24 smithi104 ceph-mon[58668]: pgmap v139: 161 pgs: 15 peering, 24 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-04-06T02:27:25.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:24 smithi017 ceph-mon[64459]: pgmap v139: 161 pgs: 15 peering, 24 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-04-06T02:27:25.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:24 smithi017 ceph-mon[67002]: pgmap v139: 161 pgs: 15 peering, 24 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-04-06T02:27:26.487 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:27:26.143+0000 7fc9f2743700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: pgmap v140: 161 pgs: 15 peering, 21 active+undersized+degraded, 25 active+undersized, 100 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/723 objects degraded (12.586%) 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: pgmap v141: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: pgmap v142: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 91/723 objects degraded (12.586%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:27.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: pgmap v140: 161 pgs: 15 peering, 21 active+undersized+degraded, 25 active+undersized, 100 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/723 objects degraded (12.586%) 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: pgmap v141: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: pgmap v142: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-04-06T02:27:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:26 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 91/723 objects degraded (12.586%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: pgmap v140: 161 pgs: 15 peering, 21 active+undersized+degraded, 25 active+undersized, 100 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 91/723 objects degraded (12.586%) 2024-04-06T02:27:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: pgmap v141: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: pgmap v142: 161 pgs: 15 peering, 11 active+undersized+degraded, 12 active+undersized, 123 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/723 objects degraded (7.746%) 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-04-06T02:27:27.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:26 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 91/723 objects degraded (12.586%), 21 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:28.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:28 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 11 pgs degraded) 2024-04-06T02:27:28.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:28 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 11 pgs degraded) 2024-04-06T02:27:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:28 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 11 pgs degraded) 2024-04-06T02:27:28.910 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:27:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:27:28.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-06T02:27:28.911 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:27:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:27:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.104:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-06T02:27:28.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:28] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.33.4" 2024-04-06T02:27:29.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:29 smithi104 ceph-mon[58668]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:27:29.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:29 smithi017 ceph-mon[64459]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:27:29.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:29 smithi017 ceph-mon[67002]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:27:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:31 smithi104 ceph-mon[58668]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-06T02:27:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:31.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[64459]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[67002]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:33.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:33 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:33.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:33 smithi104 ceph-mon[58668]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:33 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:33 smithi017 ceph-mon[67002]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:33 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:33 smithi017 ceph-mon[64459]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:35.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:35 smithi104 ceph-mon[58668]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:35.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:35 smithi017 ceph-mon[64459]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:35.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:35 smithi017 ceph-mon[67002]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:36.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:36 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:36.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:36 smithi017 ceph-mon[67002]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:36.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:36 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:36.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:36 smithi017 ceph-mon[64459]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:36.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:36 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:36.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:36 smithi104 ceph-mon[58668]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:38.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:38] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.33.4" 2024-04-06T02:27:39.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:39 smithi104 ceph-mon[58668]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:39.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:39 smithi017 ceph-mon[67002]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:39.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:39 smithi017 ceph-mon[64459]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:41.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:41 smithi104 ceph-mon[58668]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:41.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:41 smithi017 ceph-mon[67002]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:41.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:41 smithi017 ceph-mon[64459]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:42.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:42 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:42 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:42 smithi104 ceph-mon[58668]: Upgrade: osd.5 is safe to restart 2024-04-06T02:27:42.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[67002]: Upgrade: osd.5 is safe to restart 2024-04-06T02:27:42.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-06T02:27:42.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:42 smithi017 ceph-mon[64459]: Upgrade: osd.5 is safe to restart 2024-04-06T02:27:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:43 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:43 smithi104 ceph-mon[58668]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:27:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:43 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[67002]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:43.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[64459]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:27:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:27:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:43 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:27:44.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:44 smithi104 ceph-mon[58668]: Upgrade: Updating osd.5 2024-04-06T02:27:44.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:44 smithi104 ceph-mon[58668]: Deploying daemon osd.5 on smithi104 2024-04-06T02:27:44.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:44 smithi017 ceph-mon[67002]: Upgrade: Updating osd.5 2024-04-06T02:27:44.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:44 smithi017 ceph-mon[67002]: Deploying daemon osd.5 on smithi104 2024-04-06T02:27:44.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:44 smithi017 ceph-mon[64459]: Upgrade: Updating osd.5 2024-04-06T02:27:44.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:44 smithi017 ceph-mon[64459]: Deploying daemon osd.5 on smithi104 2024-04-06T02:27:45.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:45 smithi104 ceph-mon[58668]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:45.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:45 smithi017 ceph-mon[67002]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:45.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:45 smithi017 ceph-mon[64459]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:46.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:46 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:46.190 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:45 smithi104 systemd[1]: Stopping Ceph osd.5 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:27:46.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:46 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:46.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:46 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:27:46.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[32037]: 2024-04-06T02:27:46.230+0000 7fb7bdd8b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:27:46.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[32037]: 2024-04-06T02:27:46.230+0000 7fb7bdd8b700 -1 osd.5 130 *** Got signal Terminated *** 2024-04-06T02:27:46.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[32037]: 2024-04-06T02:27:46.230+0000 7fb7bdd8b700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:27:47.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:47 smithi104 ceph-mon[58668]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:47.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:47 smithi104 ceph-mon[58668]: osd.5 marked itself down and dead 2024-04-06T02:27:47.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:47 smithi017 ceph-mon[67002]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:47.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:47 smithi017 ceph-mon[67002]: osd.5 marked itself down and dead 2024-04-06T02:27:47.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:47 smithi017 ceph-mon[64459]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:27:47.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:47 smithi017 ceph-mon[64459]: osd.5 marked itself down and dead 2024-04-06T02:27:47.747 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:47 smithi104 podman[64105]: 2024-04-06 02:27:47.462249218 +0000 UTC m=+1.339477418 container died e704a0a9ebdcc41c4edf88997e0f61506c8bdca31e0305bdd8100147c4407d9e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, vcs-type=git, name=centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, ceph=True) 2024-04-06T02:27:48.055 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:47 smithi104 podman[64105]: 2024-04-06 02:27:47.872357983 +0000 UTC m=+1.749586163 container cleanup e704a0a9ebdcc41c4edf88997e0f61506c8bdca31e0305bdd8100147c4407d9e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, name=centos-stream, io.openshift.expose-services=, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-06T02:27:48.055 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:47 smithi104 bash[64105]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T02:27:48.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:48 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:48.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:48 smithi104 ceph-mon[58668]: osdmap e131: 8 total, 7 up, 8 in 2024-04-06T02:27:48.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64134]: 2024-04-06 02:27:48.124117984 +0000 UTC m=+0.662938862 container remove e704a0a9ebdcc41c4edf88997e0f61506c8bdca31e0305bdd8100147c4407d9e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, version=8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True) 2024-04-06T02:27:48.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:48 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:48.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:48 smithi017 ceph-mon[67002]: osdmap e131: 8 total, 7 up, 8 in 2024-04-06T02:27:48.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:48 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:27:48.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:48 smithi017 ceph-mon[64459]: osdmap e131: 8 total, 7 up, 8 in 2024-04-06T02:27:48.714 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64227]: 2024-04-06 02:27:48.383497624 +0000 UTC m=+0.017965853 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:48.714 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64227]: 2024-04-06 02:27:48.491754551 +0000 UTC m=+0.126222744 container create ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-06T02:27:48.714 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64227]: 2024-04-06 02:27:48.728007377 +0000 UTC m=+0.362475587 container init ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:27:48.714 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64227]: 2024-04-06 02:27:48.731690604 +0000 UTC m=+0.366158798 container start ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-06T02:27:48.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:48] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-04-06T02:27:49.009 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:48 smithi104 podman[64227]: 2024-04-06 02:27:48.819246586 +0000 UTC m=+0.453714786 container attach ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-06T02:27:49.371 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:49 smithi104 ceph-mon[58668]: pgmap v154: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:49.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:49 smithi104 ceph-mon[58668]: osdmap e132: 8 total, 7 up, 8 in 2024-04-06T02:27:49.372 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:49 smithi104 podman[64227]: 2024-04-06 02:27:49.027090386 +0000 UTC m=+0.661558591 container died ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-06T02:27:49.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:49 smithi017 ceph-mon[64459]: pgmap v154: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:49.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:49 smithi017 ceph-mon[64459]: osdmap e132: 8 total, 7 up, 8 in 2024-04-06T02:27:49.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:49 smithi017 ceph-mon[67002]: pgmap v154: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:27:49.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:49 smithi017 ceph-mon[67002]: osdmap e132: 8 total, 7 up, 8 in 2024-04-06T02:27:50.160 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:49 smithi104 podman[64252]: 2024-04-06 02:27:49.898273122 +0000 UTC m=+0.864562555 container remove ca339ce3eb587ca68797221bf97f7fafb175bd3be7595d5098abe10797078ef1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T02:27:50.160 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:49 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.5.service: Deactivated successfully. 2024-04-06T02:27:50.160 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:49 smithi104 systemd[1]: Stopped Ceph osd.5 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:27:50.161 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:49 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.5.service: Consumed 8.502s CPU time. 2024-04-06T02:27:50.161 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 systemd[1]: Starting Ceph osd.5 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:27:50.618 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 podman[64356]: 2024-04-06 02:27:50.349318805 +0000 UTC m=+0.114198942 container create dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-06T02:27:50.618 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 podman[64356]: 2024-04-06 02:27:50.253073443 +0000 UTC m=+0.017953538 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:50.618 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 podman[64356]: 2024-04-06 02:27:50.635595835 +0000 UTC m=+0.400475932 container init dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, org.label-schema.license=GPLv2) 2024-04-06T02:27:50.910 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 podman[64356]: 2024-04-06 02:27:50.639141699 +0000 UTC m=+0.404021802 container start dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-06T02:27:50.910 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:50 smithi104 podman[64356]: 2024-04-06 02:27:50.718295275 +0000 UTC m=+0.483175385 container attach dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-06T02:27:51.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-mon[58668]: pgmap v156: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 bash[64356]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-06T02:27:51.388 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-06T02:27:51.389 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T02:27:51.389 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate[64371]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-06T02:27:51.389 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 podman[64356]: 2024-04-06 02:27:51.101284112 +0000 UTC m=+0.866164214 container died dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-06T02:27:51.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:51 smithi017 ceph-mon[64459]: pgmap v156: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:27:51.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:51 smithi017 ceph-mon[67002]: pgmap v156: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:27:52.028 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 podman[64493]: 2024-04-06 02:27:51.730373685 +0000 UTC m=+0.623679616 container remove dbc4db682df992f76f51ca942acf4260f622622eef53294f1555d261e2000b17 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-06T02:27:52.028 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 podman[64558]: 2024-04-06 02:27:51.846825508 +0000 UTC m=+0.017772599 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:27:52.028 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:51 smithi104 podman[64558]: 2024-04-06 02:27:51.948294416 +0000 UTC m=+0.119241574 container create 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-06T02:27:52.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:52 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 83/723 objects degraded (11.480%), 23 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:52.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 podman[64558]: 2024-04-06 02:27:52.176428685 +0000 UTC m=+0.347375777 container init 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-06T02:27:52.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 podman[64558]: 2024-04-06 02:27:52.179615596 +0000 UTC m=+0.350562689 container start 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-06T02:27:52.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 ceph-osd[64577]: -- 172.21.15.104:0/3791201400 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5576a59a5680 con 0x5576a596f000 2024-04-06T02:27:52.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 bash[64558]: 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a 2024-04-06T02:27:52.411 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 systemd[1]: Started Ceph osd.5 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:27:52.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:52 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 83/723 objects degraded (11.480%), 23 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:52 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 83/723 objects degraded (11.480%), 23 pgs degraded (PG_DEGRADED) 2024-04-06T02:27:53.038 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:27:53.120 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:52 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:27:52.717+0000 7fbfb01cb7c0 -1 Falling back to public interface 2024-04-06T02:27:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:53 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:53 smithi104 ceph-mon[58668]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:53.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[64459]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[67002]: pgmap v157: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 66s ago 10m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 32s ago 10m - - 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (4m) 66s ago 9m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (4m) 32s ago 14m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (4m) 66s ago 16m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 66s ago 16m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (3m) 32s ago 15m 36.6M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 66s ago 15m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 66s ago 11m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:27:53.710 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (5m) 32s ago 11m 17.9M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 66s ago 14m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (2m) 66s ago 14m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (106s) 66s ago 13m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (71s) 66s ago 13m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (37s) 32s ago 13m 12.6M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 starting - - - 2992M 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (12m) 32s ago 12m 57.3M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (11m) 32s ago 11m 59.0M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (4m) 32s ago 11m 69.4M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (9m) 66s ago 9m 74.0M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (9m) 32s ago 9m 73.7M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (9m) 66s ago 9m 76.6M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:27:53.711 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (9m) 32s ago 9m 77.9M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:27:54.114 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:27:54.114 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:27:54.114 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:27:54.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 5 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 10 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:27:54.115 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:27:54.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:54 smithi104 ceph-mon[58668]: from='client.44418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:54.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:54 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3719706805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "10/23 daemons upgraded", 2024-04-06T02:27:54.453 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:27:54.454 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:27:54.454 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:27:54.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:54 smithi017 ceph-mon[67002]: from='client.44418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:54.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:54 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3719706805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:54.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:54 smithi017 ceph-mon[64459]: from='client.44418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:54.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:54 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3719706805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:27:54.861 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 83/723 objects degraded (11.480%), 23 pgs degraded 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in error state 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: osd.5 (root=default,host=smithi104) is down 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 83/723 objects degraded (11.480%), 23 pgs degraded 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-04-06T02:27:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.1f is active+undersized+degraded, acting [6,2] 2024-04-06T02:27:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: pg 5.2 is active+undersized+degraded, acting [6,0] 2024-04-06T02:27:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-04-06T02:27:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-04-06T02:27:55.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-mon[58668]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-mon[58668]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-mon[58668]: pgmap v158: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:55.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/2691523326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:55.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[67002]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[67002]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[67002]: pgmap v158: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/2691523326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[64459]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[64459]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[64459]: pgmap v158: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:55 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/2691523326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:27:55.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:27:55.416+0000 7fbfb01cb7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-06T02:27:55.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:55 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:27:55.440+0000 7fbfb01cb7c0 -1 osd.5 130 log_to_monitors true 2024-04-06T02:27:56.131 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:56 smithi104 ceph-mon[58668]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:56.132 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:56 smithi104 ceph-mon[58668]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:56.132 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:56 smithi104 ceph-mon[58668]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:56.410 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:27:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:27:56.170+0000 7fbfa1623700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:27:56.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[67002]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:56.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[67002]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:56.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[67002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:56.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[64459]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:27:56.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[64459]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:56.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:56 smithi017 ceph-mon[64459]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: pgmap v159: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: osdmap e133: 8 total, 7 up, 8 in 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:57.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:57 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:57.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: pgmap v159: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: osdmap e133: 8 total, 7 up, 8 in 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: pgmap v159: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: osdmap e133: 8 total, 7 up, 8 in 2024-04-06T02:27:57.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: from='osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:27:57.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:57.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:57 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:58.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273] boot 2024-04-06T02:27:58.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: osdmap e134: 8 total, 8 up, 8 in 2024-04-06T02:27:58.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:27:58.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:58 smithi104 ceph-mon[58668]: osdmap e135: 8 total, 8 up, 8 in 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273] boot 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: osdmap e134: 8 total, 8 up, 8 in 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[67002]: osdmap e135: 8 total, 8 up, 8 in 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: osd.5 [v2:172.21.15.104:6808/614174273,v1:172.21.15.104:6809/614174273] boot 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: osdmap e134: 8 total, 8 up, 8 in 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:27:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-mon[64459]: osdmap e135: 8 total, 8 up, 8 in 2024-04-06T02:27:58.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:27:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:27:58] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-04-06T02:27:59.265 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:27:59 smithi104 ceph-mon[58668]: pgmap v162: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:59.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:27:59 smithi017 ceph-mon[67002]: pgmap v162: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:27:59.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:27:59 smithi017 ceph-mon[64459]: pgmap v162: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-04-06T02:28:01.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: pgmap v164: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-04-06T02:28:01.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:01.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:28:01.099+0000 7fc9f2743700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: pgmap v164: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:01.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: pgmap v164: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:01.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:02.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:02 smithi104 ceph-mon[58668]: pgmap v165: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:02 smithi104 ceph-mon[58668]: pgmap v166: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:02 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:02.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:02 smithi104 ceph-mon[58668]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-06T02:28:02.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:02 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:02.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[64459]: pgmap v165: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[64459]: pgmap v166: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[64459]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[67002]: pgmap v165: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[67002]: pgmap v166: 161 pgs: 19 active+undersized+degraded, 30 active+undersized, 112 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 71/723 objects degraded (9.820%) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[67002]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-06T02:28:02.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:02 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:03 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:03.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:03 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:03.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:03 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:04.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:04 smithi104 ceph-mon[58668]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-06T02:28:04.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:04 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded) 2024-04-06T02:28:04.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:04 smithi017 ceph-mon[67002]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-06T02:28:04.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:04 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded) 2024-04-06T02:28:04.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:04 smithi017 ceph-mon[64459]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-04-06T02:28:04.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:04 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 19 pgs degraded) 2024-04-06T02:28:06.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:06 smithi104 ceph-mon[58668]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:28:06.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:06 smithi017 ceph-mon[64459]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:28:06.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:06 smithi017 ceph-mon[67002]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-06T02:28:08.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:08 smithi017 ceph-mon[64459]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:08.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:08 smithi017 ceph-mon[67002]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:08.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:08 smithi104 ceph-mon[58668]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:08.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:08] "GET /metrics HTTP/1.1" 200 38317 "" "Prometheus/2.33.4" 2024-04-06T02:28:10.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:10 smithi017 ceph-mon[64459]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:10.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:10 smithi017 ceph-mon[67002]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:10.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:10 smithi104 ceph-mon[58668]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:28:11.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:11.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:11 smithi104 ceph-mon[58668]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T02:28:11.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:11.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:11 smithi017 ceph-mon[67002]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T02:28:11.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:11.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:11 smithi017 ceph-mon[64459]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T02:28:12.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:12 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:12.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:12 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:12.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:12 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:13.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:13 smithi104 ceph-mon[58668]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:13.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:13 smithi017 ceph-mon[67002]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:13.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:13 smithi017 ceph-mon[64459]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:16 smithi104 ceph-mon[58668]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:16.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[67002]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:16.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:16.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[64459]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:16.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:16.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:17.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:17 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:17 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:17 smithi104 ceph-mon[58668]: Upgrade: osd.6 is safe to restart 2024-04-06T02:28:17.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[67002]: Upgrade: osd.6 is safe to restart 2024-04-06T02:28:17.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-06T02:28:17.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:17 smithi017 ceph-mon[64459]: Upgrade: osd.6 is safe to restart 2024-04-06T02:28:18.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:18 smithi104 ceph-mon[58668]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:18.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:18 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:18.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:18 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:28:18.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:18 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:18.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[67002]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[64459]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T02:28:18.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:18.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:18] "GET /metrics HTTP/1.1" 200 38464 "" "Prometheus/2.33.4" 2024-04-06T02:28:19.151 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:19 smithi104 ceph-mon[58668]: Upgrade: Updating osd.6 2024-04-06T02:28:19.151 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:19 smithi104 ceph-mon[58668]: Deploying daemon osd.6 on smithi104 2024-04-06T02:28:19.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:19 smithi017 ceph-mon[67002]: Upgrade: Updating osd.6 2024-04-06T02:28:19.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:19 smithi017 ceph-mon[67002]: Deploying daemon osd.6 on smithi104 2024-04-06T02:28:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:19 smithi017 ceph-mon[64459]: Upgrade: Updating osd.6 2024-04-06T02:28:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:19 smithi017 ceph-mon[64459]: Deploying daemon osd.6 on smithi104 2024-04-06T02:28:20.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:20 smithi104 ceph-mon[58668]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:20.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:20 smithi017 ceph-mon[67002]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:20.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:20 smithi017 ceph-mon[64459]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:21.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:21 smithi104 systemd[1]: Stopping Ceph osd.6 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:28:21.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[34920]: 2024-04-06T02:28:21.581+0000 7f3f4f7db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:28:21.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[34920]: 2024-04-06T02:28:21.581+0000 7f3f4f7db700 -1 osd.6 135 *** Got signal Terminated *** 2024-04-06T02:28:21.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[34920]: 2024-04-06T02:28:21.581+0000 7f3f4f7db700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:28:22.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:22 smithi104 ceph-mon[58668]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:28:22.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:22 smithi104 ceph-mon[58668]: osd.6 marked itself down and dead 2024-04-06T02:28:22.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:22 smithi017 ceph-mon[67002]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:28:22.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:22 smithi017 ceph-mon[67002]: osd.6 marked itself down and dead 2024-04-06T02:28:22.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:22 smithi017 ceph-mon[64459]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T02:28:22.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:22 smithi017 ceph-mon[64459]: osd.6 marked itself down and dead 2024-04-06T02:28:22.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:22 smithi104 podman[66387]: 2024-04-06 02:28:22.560111221 +0000 UTC m=+1.098031430 container died 679c3b97c9fc8c538e4e5a1a559be62008fb8f933c3fc4b6194f9aa38c897595 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-06T02:28:23.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:23 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:23.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:23 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:23.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:23 smithi104 ceph-mon[58668]: osdmap e136: 8 total, 7 up, 8 in 2024-04-06T02:28:23.233 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66387]: 2024-04-06 02:28:23.014680407 +0000 UTC m=+1.552600614 container cleanup 679c3b97c9fc8c538e4e5a1a559be62008fb8f933c3fc4b6194f9aa38c897595 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, distribution-scope=public, name=centos-stream, ceph=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8) 2024-04-06T02:28:23.233 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 bash[66387]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T02:28:23.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:23.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:23.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[67002]: osdmap e136: 8 total, 7 up, 8 in 2024-04-06T02:28:23.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:23.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:23.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:23 smithi017 ceph-mon[64459]: osdmap e136: 8 total, 7 up, 8 in 2024-04-06T02:28:23.660 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66404]: 2024-04-06 02:28:23.251154922 +0000 UTC m=+0.686635861 container remove 679c3b97c9fc8c538e4e5a1a559be62008fb8f933c3fc4b6194f9aa38c897595 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, release=754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-type=git, RELEASE=HEAD, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64) 2024-04-06T02:28:23.993 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66497]: 2024-04-06 02:28:23.59468114 +0000 UTC m=+0.017942116 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:28:23.993 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66497]: 2024-04-06 02:28:23.718533371 +0000 UTC m=+0.141794343 container create d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T02:28:23.993 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66497]: 2024-04-06 02:28:23.928071678 +0000 UTC m=+0.351332659 container init d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-06T02:28:23.993 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:23 smithi104 podman[66497]: 2024-04-06 02:28:23.931393228 +0000 UTC m=+0.354654208 container start d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235) 2024-04-06T02:28:23.993 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 podman[66497]: 2024-04-06 02:28:24.010841625 +0000 UTC m=+0.434102616 container attach d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-06T02:28:24.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:24 smithi104 ceph-mon[58668]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 88 op/s 2024-04-06T02:28:24.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:24 smithi104 ceph-mon[58668]: osdmap e137: 8 total, 7 up, 8 in 2024-04-06T02:28:24.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 podman[66497]: 2024-04-06 02:28:24.228258369 +0000 UTC m=+0.651519335 container died d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:28:24.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:24 smithi017 ceph-mon[67002]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 88 op/s 2024-04-06T02:28:24.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:24 smithi017 ceph-mon[67002]: osdmap e137: 8 total, 7 up, 8 in 2024-04-06T02:28:24.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:24 smithi017 ceph-mon[64459]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 88 op/s 2024-04-06T02:28:24.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:24 smithi017 ceph-mon[64459]: osdmap e137: 8 total, 7 up, 8 in 2024-04-06T02:28:25.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 podman[66523]: 2024-04-06 02:28:24.805870858 +0000 UTC m=+0.568964707 container remove d9e501fe91c3ecccee15768f0c66c9d81046da2dee5fdc088a00067cd4f14d25 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:28:25.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.6.service: Deactivated successfully. 2024-04-06T02:28:25.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 systemd[1]: Stopped Ceph osd.6 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:28:25.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:24 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.6.service: Consumed 7.820s CPU time. 2024-04-06T02:28:25.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 systemd[1]: Starting Ceph osd.6 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:28:25.229 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:28:25.575 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.161293185 +0000 UTC m=+0.018097139 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:28:25.575 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.291623039 +0000 UTC m=+0.148426966 container create 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:28:25.575 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.492973743 +0000 UTC m=+0.349777669 container init 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-06T02:28:25.575 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.496230291 +0000 UTC m=+0.353034227 container start 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-06T02:28:25.575 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.592529674 +0000 UTC m=+0.449333615 container attach 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-06T02:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 98s ago 10m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 29s ago 10m - - 2024-04-06T02:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (5m) 98s ago 9m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (4m) 29s ago 15m 451M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (5m) 98s ago 16m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 98s ago 17m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (3m) 29s ago 15m 38.3M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 98s ago 15m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 98s ago 11m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (5m) 29s ago 11m 18.3M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 98s ago 15m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (2m) 98s ago 14m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 98s ago 14m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (103s) 98s ago 13m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (69s) 29s ago 13m 45.2M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (34s) 29s ago 13m 12.7M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 7e9ae6079186 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (12m) 29s ago 12m 59.0M 2992M 17.2.0 e1d6a67b021e 679c3b97c9fc 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (12m) 29s ago 12m 59.8M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (4m) 29s ago 11m 69.5M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (10m) 98s ago 10m 74.0M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (10m) 29s ago 10m 73.8M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (9m) 98s ago 9m 76.6M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (9m) 29s ago 9m 78.0M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:28:26.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:26 smithi017 ceph-mon[67002]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-04-06T02:28:26.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:26 smithi017 ceph-mon[67002]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:26.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:26 smithi017 ceph-mon[64459]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-04-06T02:28:26.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:26 smithi017 ceph-mon[64459]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:26.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:26 smithi104 ceph-mon[58668]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-04-06T02:28:26.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:26 smithi104 ceph-mon[58668]: from='client.44457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-06T02:28:26.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 bash[66629]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate[66644]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-06T02:28:26.278 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:25 smithi104 podman[66629]: 2024-04-06 02:28:25.959872796 +0000 UTC m=+0.816676725 container died 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, ceph=True) 2024-04-06T02:28:26.309 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 6 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:26.310 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:28:26.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-06T02:28:26.311 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 11 2024-04-06T02:28:26.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:28:26.311 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:28:26.648 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:28:26.648 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:28:26.648 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:28:26.648 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:28:26.648 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "11/23 daemons upgraded", 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:28:26.649 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:28:26.978 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:26 smithi104 podman[66760]: 2024-04-06 02:28:26.680357117 +0000 UTC m=+0.715272198 container remove 25a95a74da6add30a7f264662b3435430d3794a3a3a5e7ced0c602209840fac9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-activate, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-06T02:28:26.979 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:26 smithi104 podman[66826]: 2024-04-06 02:28:26.897818648 +0000 UTC m=+0.115397170 container create 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-06T02:28:26.979 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:26 smithi104 podman[66826]: 2024-04-06 02:28:26.801083838 +0000 UTC m=+0.018662351 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in error state 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: osd.6 (root=default,host=smithi104) is down 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-06T02:28:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-04-06T02:28:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-04-06T02:28:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.1f is active+undersized+degraded, acting [5,2] 2024-04-06T02:28:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: pg 5.2 is active+undersized+degraded, acting [0,5] 2024-04-06T02:28:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-04-06T02:28:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:27 smithi104 ceph-mon[58668]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:27 smithi104 ceph-mon[58668]: from='client.54400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:27 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/4150040635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:27.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 podman[66826]: 2024-04-06 02:28:27.11724734 +0000 UTC m=+0.334825861 container init 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-06T02:28:27.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 podman[66826]: 2024-04-06 02:28:27.120393161 +0000 UTC m=+0.337971675 container start 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, ceph=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:28:27.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 ceph-osd[66845]: -- 172.21.15.104:0/3917736004 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f4cf63680 con 0x562f4cf2d000 2024-04-06T02:28:27.410 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 bash[66826]: 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 2024-04-06T02:28:27.411 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 systemd[1]: Started Ceph osd.6 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[67002]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[67002]: from='client.54400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/4150040635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[64459]: from='client.54394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[64459]: from='client.54400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:27 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/4150040635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:27.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:28:27.658+0000 7fe6753927c0 -1 Falling back to public interface 2024-04-06T02:28:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 0 B/s wr, 176 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/4124101205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:28:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:28 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:28.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 0 B/s wr, 176 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/4124101205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: pgmap v181: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 0 B/s wr, 176 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:28.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/4124101205' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:28:28.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:28.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:28.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:28] "GET /metrics HTTP/1.1" 200 38464 "" "Prometheus/2.33.4" 2024-04-06T02:28:30.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:30 smithi104 ceph-mon[58668]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:30.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:30 smithi017 ceph-mon[67002]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:30.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:30 smithi017 ceph-mon[64459]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:30.579 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:28:30.368+0000 7fe6753927c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-06T02:28:30.580 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:28:30.402+0000 7fe6753927c0 -1 osd.6 135 log_to_monitors true 2024-04-06T02:28:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:31 smithi104 ceph-mon[58668]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:31 smithi104 ceph-mon[58668]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:31.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:31.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[64459]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[64459]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[67002]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[67002]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:31.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:28:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:28:31.518+0000 7fe6627e2700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: osdmap e138: 8 total, 7 up, 8 in 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:32 smithi104 ceph-mon[58668]: from='osd.6 ' entity='osd.6' 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: osdmap e138: 8 total, 7 up, 8 in 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[67002]: from='osd.6 ' entity='osd.6' 2024-04-06T02:28:32.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: osdmap e138: 8 total, 7 up, 8 in 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:32 smithi017 ceph-mon[64459]: from='osd.6 ' entity='osd.6' 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378] boot 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: osdmap e139: 8 total, 8 up, 8 in 2024-04-06T02:28:33.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:28:33.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378] boot 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: osdmap e139: 8 total, 8 up, 8 in 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:28:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: osd.6 [v2:172.21.15.104:6816/3339510378,v1:172.21.15.104:6817/3339510378] boot 2024-04-06T02:28:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: osdmap e139: 8 total, 8 up, 8 in 2024-04-06T02:28:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T02:28:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:34 smithi104 ceph-mon[58668]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 0 B/s wr, 231 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:34 smithi104 ceph-mon[58668]: osdmap e140: 8 total, 8 up, 8 in 2024-04-06T02:28:34.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:34 smithi017 ceph-mon[64459]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 0 B/s wr, 231 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:34.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:34 smithi017 ceph-mon[64459]: osdmap e140: 8 total, 8 up, 8 in 2024-04-06T02:28:34.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:34 smithi017 ceph-mon[67002]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 0 B/s wr, 231 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:34.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:34 smithi017 ceph-mon[67002]: osdmap e140: 8 total, 8 up, 8 in 2024-04-06T02:28:36.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:36 smithi017 ceph-mon[64459]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 220 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:36.300 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:36 smithi017 ceph-mon[67002]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 220 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:36.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:36 smithi104 ceph-mon[58668]: pgmap v188: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 220 op/s; 86/723 objects degraded (11.895%) 2024-04-06T02:28:36.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:36 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:28:36.317+0000 7fc9f2743700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: pgmap v189: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: pgmap v190: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 204 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:37.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: pgmap v189: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: pgmap v190: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 204 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: pgmap v189: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: pgmap v190: 161 pgs: 11 active+undersized+degraded, 19 active+undersized, 131 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 204 op/s; 53/723 objects degraded (7.331%); 0 B/s, 3 objects/s recovering 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:38.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:38 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:38.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:38 smithi104 ceph-mon[58668]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T02:28:38.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:38 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:38.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:38.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[64459]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T02:28:38.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:38.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:38.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[67002]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-06T02:28:38.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded (PG_DEGRADED) 2024-04-06T02:28:38.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:38] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-04-06T02:28:39.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:39 smithi104 ceph-mon[58668]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s; 0 B/s, 9 objects/s recovering 2024-04-06T02:28:39.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:39 smithi017 ceph-mon[67002]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s; 0 B/s, 9 objects/s recovering 2024-04-06T02:28:39.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:39 smithi017 ceph-mon[64459]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s; 0 B/s, 9 objects/s recovering 2024-04-06T02:28:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:40 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded) 2024-04-06T02:28:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:40 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded) 2024-04-06T02:28:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:40 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/723 objects degraded (7.331%), 11 pgs degraded) 2024-04-06T02:28:41.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:41 smithi104 ceph-mon[58668]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s; 0 B/s, 7 objects/s recovering 2024-04-06T02:28:41.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:41 smithi017 ceph-mon[67002]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s; 0 B/s, 7 objects/s recovering 2024-04-06T02:28:41.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:41 smithi017 ceph-mon[64459]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s; 0 B/s, 7 objects/s recovering 2024-04-06T02:28:43.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:43 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:43.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:43 smithi017 ceph-mon[64459]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s; 0 B/s, 6 objects/s recovering 2024-04-06T02:28:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:43 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:43.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:43 smithi017 ceph-mon[67002]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s; 0 B/s, 6 objects/s recovering 2024-04-06T02:28:43.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:43 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:43.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:43 smithi104 ceph-mon[58668]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s; 0 B/s, 6 objects/s recovering 2024-04-06T02:28:45.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:45 smithi017 ceph-mon[64459]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 5 objects/s recovering 2024-04-06T02:28:45.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:45 smithi017 ceph-mon[67002]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 5 objects/s recovering 2024-04-06T02:28:45.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:45 smithi104 ceph-mon[58668]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 5 objects/s recovering 2024-04-06T02:28:46.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:46 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:46 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:46.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:46 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:46 smithi104 ceph-mon[58668]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[64459]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:46.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:46 smithi017 ceph-mon[67002]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-04-06T02:28:48.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:48] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.33.4" 2024-04-06T02:28:49.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:49 smithi017 ceph-mon[64459]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-04-06T02:28:49.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:49 smithi017 ceph-mon[67002]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-04-06T02:28:49.611 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:49 smithi104 ceph-mon[58668]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-04-06T02:28:51.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:51 smithi017 ceph-mon[67002]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:51.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:51 smithi017 ceph-mon[64459]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:51.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:51 smithi104 ceph-mon[58668]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:52.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:52 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:52.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:52 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:52.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:52 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: Upgrade: osd.7 is safe to restart 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: Upgrade: osd.7 is safe to restart 2024-04-06T02:28:53.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:28:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:53 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:53.637 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-06T02:28:53.637 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: Upgrade: osd.7 is safe to restart 2024-04-06T02:28:53.637 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:28:53.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:28:53.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:28:53.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T02:28:53.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:53 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:28:54.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:54 smithi017 ceph-mon[67002]: Upgrade: Updating osd.7 2024-04-06T02:28:54.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:54 smithi017 ceph-mon[67002]: Deploying daemon osd.7 on smithi104 2024-04-06T02:28:54.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:54 smithi017 ceph-mon[64459]: Upgrade: Updating osd.7 2024-04-06T02:28:54.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:54 smithi017 ceph-mon[64459]: Deploying daemon osd.7 on smithi104 2024-04-06T02:28:54.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:54 smithi104 ceph-mon[58668]: Upgrade: Updating osd.7 2024-04-06T02:28:54.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:54 smithi104 ceph-mon[58668]: Deploying daemon osd.7 on smithi104 2024-04-06T02:28:55.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:55 smithi104 ceph-mon[58668]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:55.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:55 smithi017 ceph-mon[67002]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:55.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:55 smithi017 ceph-mon[64459]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:56.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:56 smithi104 systemd[1]: Stopping Ceph osd.7 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:28:57.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[37838]: 2024-04-06T02:28:56.822+0000 7fd2eef0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:28:57.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[37838]: 2024-04-06T02:28:56.823+0000 7fd2eef0e700 -1 osd.7 140 *** Got signal Terminated *** 2024-04-06T02:28:57.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[37838]: 2024-04-06T02:28:56.823+0000 7fd2eef0e700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:28:57.437 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:28:57.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:57 smithi017 ceph-mon[67002]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:57.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:57 smithi017 ceph-mon[67002]: osd.7 marked itself down and dead 2024-04-06T02:28:57.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:57 smithi017 ceph-mon[64459]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:57.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:57 smithi017 ceph-mon[64459]: osd.7 marked itself down and dead 2024-04-06T02:28:57.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:57 smithi104 ceph-mon[58668]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:28:57.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:57 smithi104 ceph-mon[58668]: osd.7 marked itself down and dead 2024-04-06T02:28:57.644 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:57 smithi104 podman[68673]: 2024-04-06 02:28:57.66073667 +0000 UTC m=+0.949614925 container died c55522394c15553bc485ebc38d406f16521f9af461cc417ea7ea62700e218c85 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream) 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 2m ago 11m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 26s ago 11m - - 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (5m) 2m ago 10m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (5m) 26s ago 15m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:28:58.105 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (5m) 2m ago 17m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 2m ago 17m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (4m) 26s ago 16m 40.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 2m ago 16m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 2m ago 12m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (6m) 26s ago 12m 17.8M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (3m) 2m ago 15m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 2m ago 15m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 2m ago 14m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 2m ago 14m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (101s) 26s ago 14m 47.6M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (66s) 26s ago 13m 44.4M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 7e9ae6079186 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (31s) 26s ago 13m 12.6M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 8eda2c82cb18 2024-04-06T02:28:58.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (12m) 26s ago 12m 62.3M 2992M 17.2.0 e1d6a67b021e c55522394c15 2024-04-06T02:28:58.107 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (5m) 26s ago 12m 69.5M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:28:58.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (10m) 2m ago 10m 74.0M - 17.2.0 e1d6a67b021e fcba5a8c1977 2024-04-06T02:28:58.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (10m) 26s ago 10m 74.0M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:28:58.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (10m) 2m ago 10m 76.6M - 17.2.0 e1d6a67b021e 59319d4621f0 2024-04-06T02:28:58.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (10m) 26s ago 10m 78.2M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:28:58.206 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68673]: 2024-04-06 02:28:58.039667333 +0000 UTC m=+1.328545589 container cleanup c55522394c15553bc485ebc38d406f16521f9af461cc417ea7ea62700e218c85 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-06T02:28:58.206 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 bash[68673]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T02:28:58.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[67002]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:58.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[67002]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[67002]: osdmap e141: 8 total, 7 up, 8 in 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[67002]: osdmap e142: 8 total, 7 up, 8 in 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[64459]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[64459]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[64459]: osdmap e141: 8 total, 7 up, 8 in 2024-04-06T02:28:58.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-mon[64459]: osdmap e142: 8 total, 7 up, 8 in 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:58.527 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 7 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 12 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:28:58.528 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:28:58.639 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:58 smithi104 ceph-mon[58668]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T02:28:58.640 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:58 smithi104 ceph-mon[58668]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-06T02:28:58.640 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:58 smithi104 ceph-mon[58668]: osdmap e141: 8 total, 7 up, 8 in 2024-04-06T02:28:58.640 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:58 smithi104 ceph-mon[58668]: osdmap e142: 8 total, 7 up, 8 in 2024-04-06T02:28:58.640 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68691]: 2024-04-06 02:28:58.224347625 +0000 UTC m=+0.564195423 container remove c55522394c15553bc485ebc38d406f16521f9af461cc417ea7ea62700e218c85 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, version=8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , name=centos-stream, release=754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, ceph=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "12/23 daemons upgraded", 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:28:58.873 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:28:58.874 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:28:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:28:58] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.33.4" 2024-04-06T02:28:58.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68787]: 2024-04-06 02:28:58.560398727 +0000 UTC m=+0.017600753 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:28:58.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68787]: 2024-04-06 02:28:58.658375279 +0000 UTC m=+0.115577292 container create 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:28:58.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68787]: 2024-04-06 02:28:58.894316076 +0000 UTC m=+0.351518089 container init 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:28:58.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68787]: 2024-04-06 02:28:58.898432063 +0000 UTC m=+0.355634077 container start 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:28:59.250 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:58 smithi104 podman[68787]: 2024-04-06 02:28:58.977594885 +0000 UTC m=+0.434796905 container attach 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-06T02:28:59.250 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:59 smithi104 podman[68787]: 2024-04-06 02:28:59.197868776 +0000 UTC m=+0.655070801 container died 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in error state 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout: osd.7 (root=default,host=smithi104) is down 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-06T02:28:59.281 INFO:teuthology.orchestra.run.smithi017.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-06T02:28:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:59 smithi104 ceph-mon[58668]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:59 smithi104 ceph-mon[58668]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:59 smithi104 ceph-mon[58668]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:59 smithi104 ceph-mon[58668]: pgmap v202: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:28:59.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:28:59 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1524422326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:59.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[67002]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[67002]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[67002]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[67002]: pgmap v202: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1524422326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[64459]: from='client.44499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[64459]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[64459]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[64459]: pgmap v202: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:28:59.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:28:59 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1524422326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:00.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:59 smithi104 podman[68813]: 2024-04-06 02:28:59.806098813 +0000 UTC m=+0.603160606 container remove 3cedc0bdaedc8e13fa4f4025d59402fda528f59f9761fe43fe96f1a8ab6a32e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-06T02:29:00.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:59 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.7.service: Deactivated successfully. 2024-04-06T02:29:00.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:59 smithi104 systemd[1]: Stopped Ceph osd.7 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:29:00.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:28:59 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.7.service: Consumed 9.806s CPU time. 2024-04-06T02:29:00.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 systemd[1]: Starting Ceph osd.7 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:29:00.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-mon[58668]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:00.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/998305066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:00.491 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.151849941 +0000 UTC m=+0.017294383 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:29:00.491 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.273672181 +0000 UTC m=+0.139116595 container create 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-06T02:29:00.492 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.509489235 +0000 UTC m=+0.374933653 container init 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:29:00.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:00 smithi017 ceph-mon[64459]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:00.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:00 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/998305066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:00.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:00 smithi017 ceph-mon[67002]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:00.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:00 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/998305066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:00.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.512972977 +0000 UTC m=+0.378417393 container start 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-06T02:29:00.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.617705781 +0000 UTC m=+0.483150215 container attach 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-06T02:29:01.267 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 bash[68916]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T02:29:01.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate[68931]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-06T02:29:01.269 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:00 smithi104 podman[68916]: 2024-04-06 02:29:00.973144474 +0000 UTC m=+0.838588900 container died 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]: dispatch 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]: dispatch 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]: dispatch 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]: dispatch 2024-04-06T02:29:01.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:29:01.585 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]: dispatch 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:01.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]: dispatch 2024-04-06T02:29:01.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:01.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:01.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:01 smithi104 podman[69063]: 2024-04-06 02:29:01.604622816 +0000 UTC m=+0.626034548 container remove 59505ccce248a8dd1d86b0927885186343c579765ae06452c526631ed5e0f1ba (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-activate, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0) 2024-04-06T02:29:01.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:01 smithi104 podman[69123]: 2024-04-06 02:29:01.718929444 +0000 UTC m=+0.018217703 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:29:01.910 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:01 smithi104 podman[69123]: 2024-04-06 02:29:01.82214735 +0000 UTC m=+0.121435624 container create 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-06T02:29:02.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 podman[69123]: 2024-04-06 02:29:02.075107752 +0000 UTC m=+0.374395992 container init 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-06T02:29:02.268 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 podman[69123]: 2024-04-06 02:29:02.078157519 +0000 UTC m=+0.377445753 container start 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-06T02:29:02.269 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-osd[69142]: -- 172.21.15.104:0/472956647 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55feca71b680 con 0x55feca6e5000 2024-04-06T02:29:02.269 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 bash[69123]: 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca 2024-04-06T02:29:02.269 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 systemd[1]: Started Ceph osd.7 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]': finished 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]': finished 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]': finished 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]': finished 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]': finished 2024-04-06T02:29:02.603 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]': finished 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]': finished 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]': finished 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]': finished 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]': finished 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: osdmap e143: 8 total, 7 up, 8 in 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: osdmap e144: 8 total, 7 up, 8 in 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]': finished 2024-04-06T02:29:02.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]': finished 2024-04-06T02:29:02.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]': finished 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: osdmap e143: 8 total, 7 up, 8 in 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: osdmap e144: 8 total, 7 up, 8 in 2024-04-06T02:29:02.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 2]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [0, 7]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 3]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 7]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 7]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.16", "id": [5, 7]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 2]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 6]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 7]}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.1b"}]': finished 2024-04-06T02:29:02.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: osdmap e143: 8 total, 7 up, 8 in 2024-04-06T02:29:02.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: osdmap e144: 8 total, 7 up, 8 in 2024-04-06T02:29:02.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:02 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:02.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:02 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:29:02.619+0000 7fbfd5e397c0 -1 Falling back to public interface 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: pgmap v207: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: Health check failed: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:03 smithi104 ceph-mon[58668]: osdmap e145: 8 total, 7 up, 8 in 2024-04-06T02:29:03.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: pgmap v207: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: Health check failed: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[67002]: osdmap e145: 8 total, 7 up, 8 in 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: pgmap v207: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: Health check failed: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:03.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:03 smithi017 ceph-mon[64459]: osdmap e145: 8 total, 7 up, 8 in 2024-04-06T02:29:05.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:05 smithi104 ceph-mon[58668]: pgmap v209: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:05.586 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:29:05.368+0000 7fbfd5e397c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-06T02:29:05.586 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:05 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:29:05.403+0000 7fbfd5e397c0 -1 osd.7 140 log_to_monitors true 2024-04-06T02:29:05.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:05 smithi017 ceph-mon[67002]: pgmap v209: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:05.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:05 smithi017 ceph-mon[64459]: pgmap v209: 161 pgs: 6 remapped+peering, 26 active+undersized+degraded, 41 active+undersized, 88 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-04-06T02:29:06.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:06 smithi104 ceph-mon[58668]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:06 smithi104 ceph-mon[58668]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:06.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:06 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:06.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[64459]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[64459]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:06.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:06.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[67002]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[67002]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T02:29:06.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:06.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:06 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.160 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:29:06 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:29:06.778+0000 7fbfc3289700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:29:07.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: pgmap v210: 161 pgs: 4 active+clean+remapped, 6 remapped+peering, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%); 1/723 objects misplaced (0.138%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: osdmap e146: 8 total, 7 up, 8 in 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='osd.7 ' entity='osd.7' 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:07 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: pgmap v210: 161 pgs: 4 active+clean+remapped, 6 remapped+peering, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%); 1/723 objects misplaced (0.138%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:07.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: osdmap e146: 8 total, 7 up, 8 in 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='osd.7 ' entity='osd.7' 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: pgmap v210: 161 pgs: 4 active+clean+remapped, 6 remapped+peering, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%); 1/723 objects misplaced (0.138%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: osdmap e146: 8 total, 7 up, 8 in 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-06T02:29:07.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='osd.7 ' entity='osd.7' 2024-04-06T02:29:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:07 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:08.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[67002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:29:08.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[67002]: osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796] boot 2024-04-06T02:29:08.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[67002]: osdmap e147: 8 total, 8 up, 8 in 2024-04-06T02:29:08.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:29:08.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[64459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:29:08.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[64459]: osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796] boot 2024-04-06T02:29:08.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[64459]: osdmap e147: 8 total, 8 up, 8 in 2024-04-06T02:29:08.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:29:08.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:08 smithi104 ceph-mon[58668]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-06T02:29:08.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:08 smithi104 ceph-mon[58668]: osd.7 [v2:172.21.15.104:6824/1852614796,v1:172.21.15.104:6825/1852614796] boot 2024-04-06T02:29:08.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:08 smithi104 ceph-mon[58668]: osdmap e147: 8 total, 8 up, 8 in 2024-04-06T02:29:08.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T02:29:08.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:08] "GET /metrics HTTP/1.1" 200 38554 "" "Prometheus/2.33.4" 2024-04-06T02:29:09.455 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:09 smithi104 ceph-mon[58668]: pgmap v213: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:09.455 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:09 smithi104 ceph-mon[58668]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-06T02:29:09.455 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:09 smithi104 ceph-mon[58668]: osdmap e148: 8 total, 8 up, 8 in 2024-04-06T02:29:09.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[64459]: pgmap v213: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:09.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[64459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-06T02:29:09.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[64459]: osdmap e148: 8 total, 8 up, 8 in 2024-04-06T02:29:09.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[67002]: pgmap v213: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%); 7 B/s, 0 objects/s recovering 2024-04-06T02:29:09.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[67002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-06T02:29:09.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:09 smithi017 ceph-mon[67002]: osdmap e148: 8 total, 8 up, 8 in 2024-04-06T02:29:11.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:10 smithi104 ceph-mon[58668]: osdmap e149: 8 total, 8 up, 8 in 2024-04-06T02:29:11.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:10 smithi104 ceph-mon[58668]: pgmap v216: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%) 2024-04-06T02:29:11.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:10 smithi017 ceph-mon[64459]: osdmap e149: 8 total, 8 up, 8 in 2024-04-06T02:29:11.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:10 smithi017 ceph-mon[64459]: pgmap v216: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%) 2024-04-06T02:29:11.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:10 smithi017 ceph-mon[67002]: osdmap e149: 8 total, 8 up, 8 in 2024-04-06T02:29:11.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:10 smithi017 ceph-mon[67002]: pgmap v216: 161 pgs: 10 active+clean+remapped, 25 active+undersized+degraded, 40 active+undersized, 86 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 95/723 objects degraded (13.140%); 2/723 objects misplaced (0.277%) 2024-04-06T02:29:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:12.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: pgmap v217: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: pgmap v218: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.238 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all osd 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T02:29:12.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: pgmap v217: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: pgmap v218: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all osd 2024-04-06T02:29:12.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T02:29:12.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: Upgrade: Setting require_osd_release to 19 squid 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-06T02:29:12.242 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[67002]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: Upgrade: Setting require_osd_release to 19 squid 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-06T02:29:12.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:11 smithi017 ceph-mon[64459]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:12.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: pgmap v217: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: pgmap v218: 161 pgs: 9 active+clean+remapped, 20 active+undersized+degraded, 32 active+undersized, 100 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 73/723 objects degraded (10.097%); 1/723 objects misplaced (0.138%) 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all osd 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-06T02:29:12.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-06T02:29:12.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: Upgrade: Setting require_osd_release to 19 squid 2024-04-06T02:29:12.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-06T02:29:12.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:11 smithi104 ceph-mon[58668]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: osdmap e150: 8 total, 8 up, 8 in 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all mds 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:12.995 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: osdmap e150: 8 total, 8 up, 8 in 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all mds 2024-04-06T02:29:12.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:12 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: osdmap e150: 8 total, 8 up, 8 in 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all mds 2024-04-06T02:29:13.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:12 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:14.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:13 smithi017 ceph-mon[64459]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 88 B/s, 0 objects/s recovering 2024-04-06T02:29:14.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:13 smithi017 ceph-mon[67002]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 88 B/s, 0 objects/s recovering 2024-04-06T02:29:14.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:13 smithi104 ceph-mon[58668]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 88 B/s, 0 objects/s recovering 2024-04-06T02:29:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/723 objects degraded (10.097%), 20 pgs degraded) 2024-04-06T02:29:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: Upgrade: Updating rgw.foo.smithi017.ffhtpi (1/4) 2024-04-06T02:29:15.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[67002]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/723 objects degraded (10.097%), 20 pgs degraded) 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: Upgrade: Updating rgw.foo.smithi017.ffhtpi (1/4) 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:15.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:14 smithi017 ceph-mon[64459]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/723 objects degraded (10.097%), 20 pgs degraded) 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: Upgrade: Updating rgw.foo.smithi017.ffhtpi (1/4) 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.ffhtpi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:15.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:14 smithi104 ceph-mon[58668]: Deploying daemon rgw.foo.smithi017.ffhtpi on smithi017 2024-04-06T02:29:16.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[67002]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 71 B/s, 0 objects/s recovering 2024-04-06T02:29:16.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:16.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:16.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[64459]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 71 B/s, 0 objects/s recovering 2024-04-06T02:29:16.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:16.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:15 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:15 smithi104 ceph-mon[58668]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 71 B/s, 0 objects/s recovering 2024-04-06T02:29:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:15 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:16.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:15 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:18.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:18 smithi017 ceph-mon[67002]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-04-06T02:29:18.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:18 smithi017 ceph-mon[64459]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-04-06T02:29:18.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:18 smithi104 ceph-mon[58668]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-04-06T02:29:18.898 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:18] "GET /metrics HTTP/1.1" 200 38557 "" "Prometheus/2.33.4" 2024-04-06T02:29:19.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:19.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:19 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:19 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:19.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:19 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:19.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:19 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:20.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[64459]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 46 B/s, 0 objects/s recovering 2024-04-06T02:29:20.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:20.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:20.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:20.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[67002]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 46 B/s, 0 objects/s recovering 2024-04-06T02:29:20.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:20.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:20.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:20 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:20.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:20 smithi104 ceph-mon[58668]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 46 B/s, 0 objects/s recovering 2024-04-06T02:29:20.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:20 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:20.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:20 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.qqrgsv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:20.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:20 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:21.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[67002]: Upgrade: Updating rgw.smpl.smithi017.qqrgsv (2/4) 2024-04-06T02:29:21.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[67002]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:29:21.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:21.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[64459]: Upgrade: Updating rgw.smpl.smithi017.qqrgsv (2/4) 2024-04-06T02:29:21.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[64459]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:29:21.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:21 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:21.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:21 smithi104 ceph-mon[58668]: Upgrade: Updating rgw.smpl.smithi017.qqrgsv (2/4) 2024-04-06T02:29:21.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:21 smithi104 ceph-mon[58668]: Deploying daemon rgw.smpl.smithi017.qqrgsv on smithi017 2024-04-06T02:29:21.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:21 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:22 smithi017 ceph-mon[64459]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 36 B/s, 0 objects/s recovering 2024-04-06T02:29:22.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:22 smithi017 ceph-mon[67002]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 36 B/s, 0 objects/s recovering 2024-04-06T02:29:22.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:22 smithi104 ceph-mon[58668]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 36 B/s, 0 objects/s recovering 2024-04-06T02:29:23.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:23 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:23.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:23 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:23.663 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:23 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:24.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:24 smithi104 ceph-mon[58668]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 94 B/s wr, 138 op/s 2024-04-06T02:29:24.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:24 smithi017 ceph-mon[64459]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 94 B/s wr, 138 op/s 2024-04-06T02:29:24.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:24 smithi017 ceph-mon[67002]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 94 B/s wr, 138 op/s 2024-04-06T02:29:25.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:25 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:25 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:25 smithi104 ceph-mon[58668]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-06T02:29:25.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[64459]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-06T02:29:25.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:25.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:25 smithi017 ceph-mon[67002]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-06T02:29:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:27 smithi104 ceph-mon[58668]: Upgrade: Updating rgw.foo.smithi104.iurqzi (3/4) 2024-04-06T02:29:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:27 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:27.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:27 smithi104 ceph-mon[58668]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:29:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[64459]: Upgrade: Updating rgw.foo.smithi104.iurqzi (3/4) 2024-04-06T02:29:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:27.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[64459]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:29:27.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[67002]: Upgrade: Updating rgw.foo.smithi104.iurqzi (3/4) 2024-04-06T02:29:27.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:27.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.iurqzi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:27.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:27.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:27 smithi017 ceph-mon[67002]: Deploying daemon rgw.foo.smithi104.iurqzi on smithi104 2024-04-06T02:29:28.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:28 smithi104 ceph-mon[58668]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:28.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:28 smithi017 ceph-mon[64459]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:28.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:28 smithi017 ceph-mon[67002]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:28.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:28] "GET /metrics HTTP/1.1" 200 38557 "" "Prometheus/2.33.4" 2024-04-06T02:29:29.653 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:29:30.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:30 smithi104 ceph-mon[58668]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 2m ago 11m 24.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 24s ago 11m - - 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (6m) 2m ago 10m 53.2M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (5m) 24s ago 16m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:29:30.320 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 2m ago 18m 50.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (4m) 24s ago 16m 40.9M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (5m) 2m ago 16m 39.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 2m ago 12m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (6m) 24s ago 12m 18.2M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 2m ago 16m 46.9M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 2m ago 15m 43.7M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (3m) 2m ago 15m 42.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 2m ago 14m 14.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (2m) 24s ago 14m 51.6M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (98s) 24s ago 14m 46.3M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 7e9ae6079186 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (63s) 24s ago 13m 52.5M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 8eda2c82cb18 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (28s) 24s ago 13m 11.7M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 3972a8bf83b3 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (5m) 24s ago 12m 69.6M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 starting - - - - 2024-04-06T02:29:30.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (11m) 24s ago 11m 74.1M - 17.2.0 e1d6a67b021e ecc16c06e9d5 2024-04-06T02:29:30.322 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 starting - - - - 2024-04-06T02:29:30.322 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (11m) 24s ago 11m 78.3M - 17.2.0 e1d6a67b021e 27e475adbd8f 2024-04-06T02:29:30.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:30 smithi017 ceph-mon[67002]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:30.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:30 smithi017 ceph-mon[64459]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:29:30.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 15 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:29:30.832 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:29:31.178 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:29:31.178 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:29:31.178 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:29:31.178 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "mon", 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "osd" 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "13/23 daemons upgraded", 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading rgw daemons", 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-04-06T02:29:31.179 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:29:31.398 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.398 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.398 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:31.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:31 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3909780489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:31.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3909780489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:31 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3909780489' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:31.593 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:29:31.593 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-06T02:29:31.593 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in error state 2024-04-06T02:29:32.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:32 smithi104 ceph-mon[58668]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:32 smithi104 ceph-mon[58668]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 170 B/s wr, 264 op/s 2024-04-06T02:29:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:32 smithi104 ceph-mon[58668]: from='client.44628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:32 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/568027236' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:32.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[64459]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[64459]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 170 B/s wr, 264 op/s 2024-04-06T02:29:32.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[64459]: from='client.44628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/568027236' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[67002]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[67002]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 170 B/s wr, 264 op/s 2024-04-06T02:29:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[67002]: from='client.44628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:29:32.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:32 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/568027236' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:29:33.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:33 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:33.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:33.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:33.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:33 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:33.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.smkgdi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T02:29:33.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:33 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:34.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:34 smithi104 ceph-mon[58668]: Upgrade: Updating rgw.smpl.smithi104.smkgdi (4/4) 2024-04-06T02:29:34.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:34 smithi104 ceph-mon[58668]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:29:34.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:34 smithi104 ceph-mon[58668]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 255 B/s wr, 316 op/s 2024-04-06T02:29:34.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[67002]: Upgrade: Updating rgw.smpl.smithi104.smkgdi (4/4) 2024-04-06T02:29:34.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[67002]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:29:34.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[67002]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 255 B/s wr, 316 op/s 2024-04-06T02:29:34.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[64459]: Upgrade: Updating rgw.smpl.smithi104.smkgdi (4/4) 2024-04-06T02:29:34.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[64459]: Deploying daemon rgw.smpl.smithi104.smkgdi on smithi104 2024-04-06T02:29:34.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:34 smithi017 ceph-mon[64459]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 255 B/s wr, 316 op/s 2024-04-06T02:29:36.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:36 smithi104 ceph-mon[58668]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 170 B/s wr, 228 op/s 2024-04-06T02:29:36.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:36 smithi017 ceph-mon[67002]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 170 B/s wr, 228 op/s 2024-04-06T02:29:36.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:36 smithi017 ceph-mon[64459]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 170 B/s wr, 228 op/s 2024-04-06T02:29:37.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[67002]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 255 B/s wr, 297 op/s 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:37.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:37 smithi017 ceph-mon[64459]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 255 B/s wr, 297 op/s 2024-04-06T02:29:37.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:37.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:37 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:37.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:37 smithi104 ceph-mon[58668]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 188 KiB/s rd, 255 B/s wr, 297 op/s 2024-04-06T02:29:38.900 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:38] "GET /metrics HTTP/1.1" 200 38557 "" "Prometheus/2.33.4" 2024-04-06T02:29:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:40 smithi104 ceph-mon[58668]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 172 op/s 2024-04-06T02:29:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:40 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:40 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:40 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:40 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[67002]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 172 op/s 2024-04-06T02:29:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[64459]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 172 op/s 2024-04-06T02:29:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:40.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:40 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[67002]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 170 B/s wr, 209 op/s 2024-04-06T02:29:42.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[64459]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 170 B/s wr, 209 op/s 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:42 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:42 smithi104 ceph-mon[58668]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 170 B/s wr, 209 op/s 2024-04-06T02:29:42.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:42 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:42 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:42 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:42.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:42 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:43.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:43 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:43.340 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:43 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:43.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:43 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:44.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:44 smithi104 ceph-mon[58668]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-04-06T02:29:44.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:44 smithi017 ceph-mon[67002]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-04-06T02:29:44.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:44 smithi017 ceph-mon[64459]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-04-06T02:29:45.877 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: Detected new or changed devices on smithi104 2024-04-06T02:29:45.877 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: Detected new or changed devices on smithi017 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 145 op/s 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 103 B/s wr, 174 op/s 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all rgw 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]': finished 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]': finished 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]': finished 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]: dispatch 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]': finished 2024-04-06T02:29:45.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: Detected new or changed devices on smithi104 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.880 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: Detected new or changed devices on smithi017 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 145 op/s 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 103 B/s wr, 174 op/s 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.881 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all rgw 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]': finished 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]': finished 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]': finished 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]': finished 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.882 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.883 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:45.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: Detected new or changed devices on smithi104 2024-04-06T02:29:45.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: Detected new or changed devices on smithi017 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 145 op/s 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 103 B/s wr, 174 op/s 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all rgw 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.ffhtpi"}]': finished 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi104.iurqzi"}]': finished 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.qqrgsv"}]': finished 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi104.smkgdi"}]': finished 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all rbd-mirror 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:29:45.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: Upgrade: Setting container_image for all ceph-exporter 2024-04-06T02:29:45.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:45.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:29:47.837 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: Upgrade: Updating iscsi.foo.smithi017.kcchvr 2024-04-06T02:29:47.837 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[67002]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: Upgrade: Updating iscsi.foo.smithi017.kcchvr 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:29:47.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:47 smithi017 ceph-mon[64459]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:47.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: Upgrade: Updating iscsi.foo.smithi017.kcchvr 2024-04-06T02:29:47.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:29:47.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.kcchvr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T02:29:47.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:29:47.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: Deploying daemon iscsi.foo.smithi017.kcchvr on smithi017 2024-04-06T02:29:47.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:47 smithi104 ceph-mon[58668]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:48.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:48] "GET /metrics HTTP/1.1" 200 38624 "" "Prometheus/2.33.4" 2024-04-06T02:29:50.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:49 smithi017 ceph-mon[64459]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:50.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:49 smithi017 ceph-mon[67002]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:50.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:49 smithi104 ceph-mon[58668]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T02:29:52.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:51 smithi017 ceph-mon[64459]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 48 op/s 2024-04-06T02:29:52.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:51 smithi017 ceph-mon[67002]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 48 op/s 2024-04-06T02:29:52.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:51 smithi104 ceph-mon[58668]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 48 op/s 2024-04-06T02:29:53.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:52 smithi017 ceph-mon[64459]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:53.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:52 smithi017 ceph-mon[67002]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:53.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:52 smithi104 ceph-mon[58668]: from='client.25276 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:29:54.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:53 smithi017 ceph-mon[67002]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:29:54.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:53 smithi017 ceph-mon[64459]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:29:54.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:53 smithi104 ceph-mon[58668]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T02:29:56.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:55 smithi017 ceph-mon[64459]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:29:56.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:55 smithi017 ceph-mon[67002]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:29:56.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:55 smithi104 ceph-mon[58668]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:29:58.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:57 smithi017 ceph-mon[67002]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:29:58.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:57 smithi017 ceph-mon[64459]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:29:58.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:57 smithi104 ceph-mon[58668]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:29:58.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:29:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:29:58] "GET /metrics HTTP/1.1" 200 38624 "" "Prometheus/2.33.4" 2024-04-06T02:30:00.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:29:59 smithi017 ceph-mon[67002]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:00.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:29:59 smithi017 ceph-mon[64459]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:00.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:29:59 smithi104 ceph-mon[58668]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:00.420 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:30:00.108+0000 7fc9c2edb700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:30:01.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:30:01.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:30:01.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:01.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T02:30:01.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:01.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:01.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:30:01.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:30:01.174 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:01.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:01.959 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:30:02.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:01 smithi104 ceph-mon[58668]: Health check failed: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-06T02:30:02.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-06T02:30:02.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]: dispatch 2024-04-06T02:30:02.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:01 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]: dispatch 2024-04-06T02:30:02.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[64459]: Health check failed: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-06T02:30:02.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-06T02:30:02.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]: dispatch 2024-04-06T02:30:02.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]: dispatch 2024-04-06T02:30:02.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[67002]: Health check failed: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-06T02:30:02.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-06T02:30:02.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]: dispatch 2024-04-06T02:30:02.237 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:01 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]: dispatch 2024-04-06T02:30:02.326 INFO:teuthology.orchestra.run.smithi017.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed." 2024-04-06T02:30:02.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch ps' 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]': finished 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]': finished 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:30:03.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[67002]: osdmap e151: 8 total, 8 up, 8 in 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]': finished 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]': finished 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:30:03.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:03 smithi017 ceph-mon[64459]: osdmap e151: 8 total, 8 up, 8 in 2024-04-06T02:30:03.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-06T02:30:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.10"}]': finished 2024-04-06T02:30:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "5.e"}]': finished 2024-04-06T02:30:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:30:03.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:03 smithi104 ceph-mon[58668]: osdmap e151: 8 total, 8 up, 8 in 2024-04-06T02:30:04.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:04 smithi104 ceph-mon[58668]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:04.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:04 smithi104 ceph-mon[58668]: osdmap e152: 8 total, 8 up, 8 in 2024-04-06T02:30:04.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:04 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:04.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:04 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:04.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[64459]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:04.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[64459]: osdmap e152: 8 total, 8 up, 8 in 2024-04-06T02:30:04.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:04.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:04.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[67002]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:04.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[67002]: osdmap e152: 8 total, 8 up, 8 in 2024-04-06T02:30:04.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:04.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:04 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:05.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:05 smithi104 ceph-mon[58668]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:05.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:05 smithi017 ceph-mon[67002]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:05.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:05 smithi017 ceph-mon[64459]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:05.468 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:30:05.468 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 25s ago 12m 23.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:30:05.468 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 1s ago 12m - - 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 running (6m) 25s ago 11m 63.0M - 3.5 e1d6a67b021e 599061aef9b7 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (6m) 1s ago 17m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (7m) 25s ago 18m 513M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (6m) 25s ago 18m 61.0M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (5m) 1s ago 17m 43.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (5m) 25s ago 17m 46.2M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 25s ago 13m 18.9M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (7m) 1s ago 13m 17.9M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 25s ago 16m 57.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (4m) 25s ago 16m 59.1M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (3m) 25s ago 15m 52.3M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (3m) 25s ago 15m 67.1M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:30:05.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (2m) 1s ago 15m 58.7M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (2m) 1s ago 14m 51.1M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 7e9ae6079186 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (98s) 1s ago 14m 59.0M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 8eda2c82cb18 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (63s) 1s ago 14m 58.8M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 3972a8bf83b3 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (6m) 1s ago 13m 69.6M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (47s) 25s ago 11m 76.4M - 19.0.0-2027-g9f192350 a5511c7deba9 637b35122486 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (35s) 1s ago 11m 76.4M - 19.0.0-2027-g9f192350 a5511c7deba9 260e3587d99f 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (41s) 25s ago 11m 76.3M - 19.0.0-2027-g9f192350 a5511c7deba9 01115783b119 2024-04-06T02:30:05.470 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (29s) 1s ago 11m 77.7M - 19.0.0-2027-g9f192350 a5511c7deba9 c2b10f0dce8c 2024-04-06T02:30:06.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions' 2024-04-06T02:30:07.409 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:07 smithi104 ceph-mon[58668]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:07.410 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:07 smithi104 ceph-mon[58668]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:07.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:07 smithi017 ceph-mon[64459]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:07.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:07 smithi017 ceph-mon[64459]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:07.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:07 smithi017 ceph-mon[67002]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:30:07.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:07 smithi017 ceph-mon[67002]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 283 B/s rd, 0 op/s 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:30:08.409 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 17 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:30:08.410 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:30:08.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:08] "GET /metrics HTTP/1.1" 200 38624 "" "Prometheus/2.33.4" 2024-04-06T02:30:09.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-06T02:30:09.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:30:09.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: Checking dashboard <-> RGW credentials 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1777061394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: Checking dashboard <-> RGW credentials 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1777061394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.661 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: Checking dashboard <-> RGW credentials 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1777061394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:30:09.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:30:09.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[64459]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[64459]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[64459]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[67002]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[67002]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.418 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:10 smithi017 ceph-mon[67002]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:30:10.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:10 smithi104 ceph-mon[58668]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:10 smithi104 ceph-mon[58668]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:10.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:10 smithi104 ceph-mon[58668]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-06T02:30:10.938 INFO:teuthology.orchestra.run.smithi017.stdout:wait for servicemap items w/ changing names to refresh 2024-04-06T02:30:11.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'sleep 60' 2024-04-06T02:30:12.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:12 smithi017 ceph-mon[67002]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:12.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:12 smithi017 ceph-mon[64459]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:12.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:12 smithi104 ceph-mon[58668]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:13.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3614478362' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:30:13.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/279205918' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:13.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3614478362' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:30:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/279205918' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:13.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:13 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:13.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:13 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3614478362' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T02:30:13.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:13 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/279205918' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:13.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:13 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]: dispatch 2024-04-06T02:30:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[64459]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5 op/s 2024-04-06T02:30:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]': finished 2024-04-06T02:30:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[64459]: osdmap e153: 8 total, 8 up, 8 in 2024-04-06T02:30:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3944804754' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:14.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:14.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[67002]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5 op/s 2024-04-06T02:30:14.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]': finished 2024-04-06T02:30:14.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[67002]: osdmap e153: 8 total, 8 up, 8 in 2024-04-06T02:30:14.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3944804754' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:14.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:14 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:14.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:14 smithi104 ceph-mon[58668]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5 op/s 2024-04-06T02:30:14.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:14 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/597518159"}]': finished 2024-04-06T02:30:14.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:14 smithi104 ceph-mon[58668]: osdmap e153: 8 total, 8 up, 8 in 2024-04-06T02:30:14.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:14 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3944804754' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:14.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:14 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]: dispatch 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]': finished 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[64459]: osdmap e154: 8 total, 8 up, 8 in 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]: dispatch 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]': finished 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[67002]: osdmap e154: 8 total, 8 up, 8 in 2024-04-06T02:30:15.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:15 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]: dispatch 2024-04-06T02:30:15.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:15 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3359958585"}]': finished 2024-04-06T02:30:15.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:15 smithi104 ceph-mon[58668]: osdmap e154: 8 total, 8 up, 8 in 2024-04-06T02:30:15.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:15 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]: dispatch 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]': finished 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: osdmap e155: 8 total, 8 up, 8 in 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:16.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]: dispatch 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]': finished 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: osdmap e155: 8 total, 8 up, 8 in 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:16.487 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:16 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]: dispatch 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3639493881' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/812144422"}]': finished 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: osdmap e155: 8 total, 8 up, 8 in 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:16 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]: dispatch 2024-04-06T02:30:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:17 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]': finished 2024-04-06T02:30:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:17 smithi104 ceph-mon[58668]: osdmap e156: 8 total, 8 up, 8 in 2024-04-06T02:30:17.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:17 smithi104 ceph-mon[58668]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:17.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:17 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1153201442' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:17.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:17 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]': finished 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[67002]: osdmap e156: 8 total, 8 up, 8 in 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[67002]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1153201442' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:17.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/974381802' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1624761725"}]': finished 2024-04-06T02:30:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[64459]: osdmap e156: 8 total, 8 up, 8 in 2024-04-06T02:30:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[64459]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-06T02:30:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1153201442' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:17.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:17 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]: dispatch 2024-04-06T02:30:18.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:18] "GET /metrics HTTP/1.1" 200 38702 "" "Prometheus/2.33.4" 2024-04-06T02:30:18.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]': finished 2024-04-06T02:30:18.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[67002]: osdmap e157: 8 total, 8 up, 8 in 2024-04-06T02:30:18.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3405055326' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:18.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:18.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]': finished 2024-04-06T02:30:18.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[64459]: osdmap e157: 8 total, 8 up, 8 in 2024-04-06T02:30:18.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3405055326' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:18.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:18 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:18 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2144555979"}]': finished 2024-04-06T02:30:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:18 smithi104 ceph-mon[58668]: osdmap e157: 8 total, 8 up, 8 in 2024-04-06T02:30:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:18 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3405055326' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:18.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:18 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]: dispatch 2024-04-06T02:30:19.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[67002]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]': finished 2024-04-06T02:30:19.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[67002]: osdmap e158: 8 total, 8 up, 8 in 2024-04-06T02:30:19.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[67002]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:19.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[64459]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]': finished 2024-04-06T02:30:19.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[64459]: osdmap e158: 8 total, 8 up, 8 in 2024-04-06T02:30:19.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:19 smithi017 ceph-mon[64459]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:19.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:19 smithi104 ceph-mon[58668]: from='client.? ' entity='client.iscsi.foo.smithi017.kcchvr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/597518159"}]': finished 2024-04-06T02:30:19.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:19 smithi104 ceph-mon[58668]: osdmap e158: 8 total, 8 up, 8 in 2024-04-06T02:30:19.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:19 smithi104 ceph-mon[58668]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:21.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:21 smithi104 ceph-mon[58668]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s 2024-04-06T02:30:21.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:21 smithi017 ceph-mon[67002]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s 2024-04-06T02:30:21.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:21 smithi017 ceph-mon[64459]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s 2024-04-06T02:30:22.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:22 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:22.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:22 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:22.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:22 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:23.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:23 smithi104 ceph-mon[58668]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:30:23.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:23 smithi017 ceph-mon[67002]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:30:23.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:23 smithi017 ceph-mon[64459]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T02:30:25.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:25 smithi104 ceph-mon[58668]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:25.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:25 smithi017 ceph-mon[67002]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:25.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:25 smithi017 ceph-mon[64459]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:27.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:27 smithi104 ceph-mon[58668]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-06T02:30:27.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:27 smithi017 ceph-mon[67002]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-06T02:30:27.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:27 smithi017 ceph-mon[64459]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-06T02:30:28.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:28] "GET /metrics HTTP/1.1" 200 38702 "" "Prometheus/2.33.4" 2024-04-06T02:30:29.887 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:29 smithi017 ceph-mon[67002]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-06T02:30:29.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:29 smithi017 ceph-mon[64459]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-06T02:30:29.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:29 smithi104 ceph-mon[58668]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-06T02:30:30.901 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:30 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:30.901 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:30 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:30.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:30 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:31.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:31 smithi104 ceph-mon[58668]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:31.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:31 smithi017 ceph-mon[67002]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:31.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:31 smithi017 ceph-mon[64459]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:32.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:32 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:32.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:32 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:32.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:32 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:33.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:33 smithi104 ceph-mon[58668]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:33.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:33 smithi017 ceph-mon[67002]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:33.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:33 smithi017 ceph-mon[64459]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:35.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:35 smithi104 ceph-mon[58668]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:35.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:35 smithi017 ceph-mon[67002]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:35.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:35 smithi017 ceph-mon[64459]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:37.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:37 smithi104 ceph-mon[58668]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:37.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:37 smithi017 ceph-mon[67002]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:37.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:37 smithi017 ceph-mon[64459]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:38.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:38] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.33.4" 2024-04-06T02:30:39.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:39 smithi104 ceph-mon[58668]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:39.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:39 smithi017 ceph-mon[67002]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:39.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:39 smithi017 ceph-mon[64459]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:41.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:41 smithi104 ceph-mon[58668]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:41.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:41 smithi017 ceph-mon[67002]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:41.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:41 smithi017 ceph-mon[64459]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:42.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:42 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:42.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:42 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:42.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:42 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:43.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:43 smithi104 ceph-mon[58668]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:43.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:43 smithi017 ceph-mon[67002]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:43.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:43 smithi017 ceph-mon[64459]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:45.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:45 smithi104 ceph-mon[58668]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:45.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:45 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:45.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:45 smithi017 ceph-mon[64459]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:45.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:45 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:45.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:45 smithi017 ceph-mon[67002]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:45.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:45 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:30:47.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:47 smithi104 ceph-mon[58668]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:47.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:47 smithi017 ceph-mon[67002]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:47.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:47 smithi017 ceph-mon[64459]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:48.626 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:48] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.33.4" 2024-04-06T02:30:49.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:49 smithi017 ceph-mon[64459]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:49.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:49 smithi017 ceph-mon[67002]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:50.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:49 smithi104 ceph-mon[58668]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:51.961 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:51 smithi104 ceph-mon[58668]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:51.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:51 smithi017 ceph-mon[67002]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:51.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:51 smithi017 ceph-mon[64459]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:52.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:52 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:52.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:52 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:53.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:52 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:30:53.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:53 smithi017 ceph-mon[67002]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:53.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:53 smithi017 ceph-mon[64459]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:54.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:53 smithi104 ceph-mon[58668]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:55.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:55 smithi017 ceph-mon[67002]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:55.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:55 smithi017 ceph-mon[64459]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:56.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:55 smithi104 ceph-mon[58668]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:57.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:57 smithi017 ceph-mon[67002]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:57.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:57 smithi017 ceph-mon[64459]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:58.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:57 smithi104 ceph-mon[58668]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:30:58.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:30:58 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:30:58] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.33.4" 2024-04-06T02:30:59.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:30:59 smithi017 ceph-mon[67002]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:30:59.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:30:59 smithi017 ceph-mon[64459]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:00.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:30:59 smithi104 ceph-mon[58668]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:00.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:00 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:00.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:00 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:01.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:00 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:01.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:01 smithi017 ceph-mon[67002]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:01.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:01 smithi017 ceph-mon[64459]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:02.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:01 smithi104 ceph-mon[58668]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:02.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:02 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:02.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:02 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:02.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:02 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:03.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:03 smithi017 ceph-mon[67002]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:03.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:03 smithi017 ceph-mon[64459]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:04.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:03 smithi104 ceph-mon[58668]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:05.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:05 smithi017 ceph-mon[67002]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:05.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:05 smithi017 ceph-mon[64459]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:06.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:05 smithi104 ceph-mon[58668]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:07.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:07 smithi017 ceph-mon[67002]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:07.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:07 smithi017 ceph-mon[64459]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:08.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:07 smithi104 ceph-mon[58668]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:08.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:08 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:31:08.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:08 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:31:08.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:31:08] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.33.4" 2024-04-06T02:31:09.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:08 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:09.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:09.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:09 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:31:10.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:10.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:31:10.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:31:10.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-06T02:31:10.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:10.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:09 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' 2024-04-06T02:31:11.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:11 smithi104 ceph-mon[58668]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-06T02:31:11.739 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:11 smithi017 ceph-mon[67002]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-06T02:31:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:11 smithi017 ceph-mon[64459]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-06T02:31:12.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:12 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:12.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:12 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:12.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:12 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:13.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:13 smithi104 ceph-mon[58668]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:13.660 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:13 smithi017 ceph-mon[67002]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:13.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:13 smithi017 ceph-mon[64459]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:14.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch ps' 2024-04-06T02:31:15.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:15 smithi104 ceph-mon[58668]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:15.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:15 smithi017 ceph-mon[67002]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:15.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:15 smithi017 ceph-mon[64459]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:16.514 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (8m) 68s ago 13m 23.5M - 0.23.0 ba2b418f427c cbef90f04e78 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi104 *:3000 error 73s ago 13m - - 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.kcchvr smithi017 unknown 68s ago 12m - - 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi104 *:8443,9283,8765 running (7m) 73s ago 18m 452M - 19.0.0-2027-g9f192350 a5511c7deba9 9b973d332e81 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (8m) 68s ago 19m 515M - 19.0.0-2027-g9f192350 a5511c7deba9 fc819800a373 2024-04-06T02:31:16.533 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (7m) 68s ago 19m 61.9M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 6255652404a1 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi104 running (6m) 73s ago 18m 43.1M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 bfeac07a24d9 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (6m) 68s ago 18m 45.4M 2048M 19.0.0-2027-g9f192350 a5511c7deba9 5525a61ec5f7 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (9m) 68s ago 14m 19.4M - 1.3.1 1dbe0e931976 f9dfc8b2fd49 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi104 *:9100 running (8m) 73s ago 14m 17.9M - 1.3.1 1dbe0e931976 e14e1595bb23 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (5m) 68s ago 17m 58.0M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 d8143254bdeb 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (5m) 68s ago 17m 58.2M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 b86017a8507a 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (5m) 68s ago 17m 51.8M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 0335ae11636f 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (4m) 68s ago 16m 67.4M 2737M 19.0.0-2027-g9f192350 a5511c7deba9 85c4daa4b795 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi104 running (3m) 73s ago 16m 58.7M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 6cfd45ba23db 2024-04-06T02:31:16.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi104 running (3m) 73s ago 16m 51.1M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 7e9ae6079186 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi104 running (2m) 73s ago 15m 59.0M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 8eda2c82cb18 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi104 running (2m) 73s ago 15m 58.8M 2992M 19.0.0-2027-g9f192350 a5511c7deba9 3972a8bf83b3 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi104 *:9095 running (7m) 73s ago 14m 69.6M - 2.33.4 514e6a882f6e 9498923da881 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ffhtpi smithi017 *:8000 running (118s) 68s ago 12m 76.6M - 19.0.0-2027-g9f192350 a5511c7deba9 637b35122486 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi104.iurqzi smithi104 *:8000 running (106s) 73s ago 12m 76.4M - 19.0.0-2027-g9f192350 a5511c7deba9 260e3587d99f 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.qqrgsv smithi017 *:80 running (112s) 68s ago 12m 76.6M - 19.0.0-2027-g9f192350 a5511c7deba9 01115783b119 2024-04-06T02:31:16.535 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi104.smkgdi smithi104 *:80 running (100s) 73s ago 12m 77.7M - 19.0.0-2027-g9f192350 a5511c7deba9 c2b10f0dce8c 2024-04-06T02:31:16.660 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:16 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:16.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:16 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:16.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:16 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:17.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions' 2024-04-06T02:31:17.538 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:17 smithi017 ceph-mon[67002]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:17.538 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:17 smithi017 ceph-mon[67002]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:17.538 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:17 smithi017 ceph-mon[64459]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:17.538 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:17 smithi017 ceph-mon[64459]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:17.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:17 smithi104 ceph-mon[58668]: from='client.54715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:17.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:17 smithi104 ceph-mon[58668]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T02:31:18.844 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:18 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:31:18] "GET /metrics HTTP/1.1" 200 38706 "" "Prometheus/2.33.4" 2024-04-06T02:31:19.390 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:31:19.390 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-04-06T02:31:19.390 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 3 2024-04-06T02:31:19.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 2 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 8 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 4 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)": 17 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-04-06T02:31:19.391 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:31:19.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:19 smithi104 ceph-mon[58668]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:31:19.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:19 smithi017 ceph-mon[64459]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:31:19.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:19 smithi017 ceph-mon[67002]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T02:31:20.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch upgrade status' 2024-04-06T02:31:20.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:20 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/1598738154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:20.661 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:20 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/1598738154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:20.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:20 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/1598738154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:21.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:21 smithi104 ceph-mon[58668]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:21.660 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:21 smithi017 ceph-mon[67002]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:21.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:21 smithi017 ceph-mon[64459]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:22.125 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw", 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-04-06T02:31:22.126 INFO:teuthology.orchestra.run.smithi017.stdout: "mon", 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout: "osd" 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "17/23 daemons upgraded", 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed.", 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": true 2024-04-06T02:31:22.127 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:31:22.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:22 smithi104 ceph-mon[58668]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:22.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:22 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:22.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:22 smithi017 ceph-mon[64459]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:22.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:22 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:22.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:22 smithi017 ceph-mon[67002]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:22.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:22 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:22.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph health detail' 2024-04-06T02:31:23.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:23 smithi104 ceph-mon[58668]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:23.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:23 smithi017 ceph-mon[67002]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:23.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:23 smithi017 ceph-mon[64459]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:24.827 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:31:24.827 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-06T02:31:24.827 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.kcchvr on smithi017 is in unknown state 2024-04-06T02:31:24.827 INFO:teuthology.orchestra.run.smithi017.stdout: daemon grafana.a on smithi104 is in error state 2024-04-06T02:31:24.827 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.kcchvr on host smithi017 failed. 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrade daemon: iscsi.foo.smithi017.kcchvr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.kcchvr ... 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Creating ceph-iscsi config... 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/iscsi-gateway.cfg 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/iscsi.foo.smithi017.kcchvr/tcmu-runner-entrypoint.sh 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2ddbfad14a\x2df3ba\x2d11ee\x2db649\x2dcb9ed24678a4.slice/ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl start ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:Traceback (most recent call last): 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: return _run_code(code, main_globals, None, 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: exec(code, run_globals) 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5534, in 2024-04-06T02:31:24.828 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 5522, in main 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3045, in command_deploy_from 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3080, in _common_deploy 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 3100, in _deploy_daemon_container 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1036, in deploy_daemon 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/__main__.py", line 1172, in deploy_daemon_units 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/cephadm.6b910254142a195ed275d8e160abbf7ad9d749551a85757e6f4fb9b0105d7af0/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout:RuntimeError: Failed command: systemctl start ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr: Job for ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service failed because the control process exited with error code. 2024-04-06T02:31:24.829 INFO:teuthology.orchestra.run.smithi017.stdout:See "systemctl status ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" and "journalctl -xeu ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@iscsi.foo.smithi017.kcchvr.service" for details. 2024-04-06T02:31:25.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:24 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/3594063156' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:31:25.235 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:24 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/3594063156' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:31:25.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:24 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/3594063156' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-06T02:31:25.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-06T02:31:25.986 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:25 smithi017 ceph-mon[67002]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:25.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:25 smithi017 ceph-mon[64459]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:26.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:25 smithi104 ceph-mon[58668]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:27.526 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-04-06T02:31:28.138 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:27 smithi017 ceph-mon[67002]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:28.139 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:27 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/4014695168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:28.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:27 smithi017 ceph-mon[64459]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:28.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:27 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/4014695168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:28.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:27 smithi104 ceph-mon[58668]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:28.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:27 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/4014695168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:28.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-06T02:31:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:28 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:31:28] "GET /metrics HTTP/1.1" 200 38706 "" "Prometheus/2.33.4" 2024-04-06T02:31:29.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:29 smithi017 ceph-mon[67002]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:29.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:29 smithi017 ceph-mon[64459]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:29.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:29 smithi104 ceph-mon[58668]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:30.297 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)" 2024-04-06T02:31:30.561 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:30 smithi017 ceph-mon[67002]: from='client.? 172.21.15.17:0/793214628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:30.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:30 smithi017 ceph-mon[64459]: from='client.? 172.21.15.17:0/793214628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:30.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:30 smithi104 ceph-mon[58668]: from='client.? 172.21.15.17:0/793214628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:31:30.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-06T02:31:31.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:31 smithi017 ceph-mon[64459]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:31.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:31 smithi017 ceph-mon[64459]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:31.522 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:31 smithi017 ceph-mon[67002]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:31.522 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:31 smithi017 ceph-mon[67002]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:31.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:31 smithi104 ceph-mon[58668]: from='mgr.44106 172.21.15.17:0/3710449447' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:31:31.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:31 smithi104 ceph-mon[58668]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:32.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:32 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:32.653 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:32 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:32.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:32 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:33.117 INFO:teuthology.orchestra.run.smithi017.stdout:osd 8 89s ago - 2024-04-06T02:31:33.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:33 smithi104 ceph-mon[58668]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:33.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:33 smithi104 ceph-mon[58668]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:33.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:33 smithi017 ceph-mon[64459]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:33.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:33 smithi017 ceph-mon[64459]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:33.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:33 smithi017 ceph-mon[67002]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:33.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:33 smithi017 ceph-mon[67002]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:33.779 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:31:33.788 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-04-06T02:31:33.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch upgrade ls' 2024-04-06T02:31:35.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:35 smithi017 ceph-mon[67002]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:35.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:35 smithi017 ceph-mon[64459]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:35.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:35 smithi104 ceph-mon[58668]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:36.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:36 smithi104 ceph-mon[58668]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:36.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:36 smithi017 ceph-mon[67002]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:36.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:36 smithi017 ceph-mon[64459]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout: "image": "quay.io/ceph/ceph", 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout: "registry": "quay.io", 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout: "bare_image": "ceph/ceph", 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout: "versions": [] 2024-04-06T02:31:37.570 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-04-06T02:31:37.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:37 smithi104 ceph-mon[58668]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:37.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:37 smithi017 ceph-mon[64459]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:37.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:37 smithi017 ceph-mon[67002]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:38.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-06T02:31:38.752 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:38 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:31:38] "GET /metrics HTTP/1.1" 200 38701 "" "Prometheus/2.33.4" 2024-04-06T02:31:39.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:39 smithi017 ceph-mon[67002]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:39.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:39 smithi017 ceph-mon[64459]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:39.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:39 smithi104 ceph-mon[58668]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:41.191 INFO:teuthology.orchestra.run.smithi017.stdout: "16.2.0", 2024-04-06T02:31:41.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:41 smithi104 ceph-mon[58668]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:41.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:41 smithi104 ceph-mon[58668]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:41.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:41 smithi017 ceph-mon[64459]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:41.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:41 smithi017 ceph-mon[64459]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:41.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:41 smithi017 ceph-mon[67002]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:41.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:41 smithi017 ceph-mon[67002]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:41.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-06T02:31:42.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:42 smithi104 ceph-mon[58668]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:42.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:42 smithi017 ceph-mon[64459]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:42.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:42 smithi017 ceph-mon[67002]: from='client.54664 -' entity='client.iscsi.foo.smithi017.kcchvr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T02:31:43.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:43 smithi104 ceph-mon[58668]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:43.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:43 smithi017 ceph-mon[64459]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:43.683 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:43 smithi017 ceph-mon[67002]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T02:31:44.546 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2", 2024-04-06T02:31:44.546 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2-20210505", 2024-04-06T02:31:44.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:44 smithi104 ceph-mon[58668]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:44.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:44 smithi017 ceph-mon[64459]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:44.736 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:44 smithi017 ceph-mon[67002]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:31:45.224 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-06T02:31:45.233 INFO:tasks.cephadm:Teardown begin 2024-04-06T02:31:45.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:31:45.265 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:31:45.301 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-06T02:31:45.301 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T02:31:45.322 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T02:31:45.358 INFO:tasks.cephadm:Stopping all daemons... 2024-04-06T02:31:45.358 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-06T02:31:45.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a 2024-04-06T02:31:45.486 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-mon[67002]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:45.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-mon[64459]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:45.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:45 smithi017 systemd[1]: Stopping Ceph mon.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:45.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:45 smithi104 ceph-mon[58668]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T02:31:45.985 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[64455]: 2024-04-06T02:31:45.708+0000 7f16a6734700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a[64455]: 2024-04-06T02:31:45.708+0000 7f16a6734700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:45 smithi017 podman[90903]: 2024-04-06 02:31:45.801306647 +0000 UTC m=+0.229966322 container died 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235) 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE Bus STOPPING 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE Bus STOPPED 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE Bus STARTING 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: CherryPy Checker: 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: The Application mounted at '' has an empty config. 2024-04-06T02:31:45.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:31:45.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE Serving on http://:::9283 2024-04-06T02:31:45.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:45 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: [06/Apr/2024:02:31:45] ENGINE Bus STARTED 2024-04-06T02:31:46.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:46 smithi017 podman[90903]: 2024-04-06 02:31:46.151075692 +0000 UTC m=+0.579735354 container cleanup 6255652404a1e210a621dd1ffc663d3343f333fd80c7edc9589d1f504c068a66 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-06T02:31:46.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Apr 06 02:31:46 smithi017 bash[90903]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-a 2024-04-06T02:31:46.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.a.service' 2024-04-06T02:31:46.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:31:46.623 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-06T02:31:46.623 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-06T02:31:46.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.c 2024-04-06T02:31:46.669 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:46 smithi017 systemd[1]: Stopping Ceph mon.c for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:47.235 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c[66998]: 2024-04-06T02:31:46.955+0000 7fa85555f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:31:47.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:46 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c[66998]: 2024-04-06T02:31:46.955+0000 7fa85555f700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-06T02:31:47.236 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:47 smithi017 podman[91034]: 2024-04-06 02:31:47.161127696 +0000 UTC m=+0.333062385 container died 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-06T02:31:47.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:47 smithi017 podman[91034]: 2024-04-06 02:31:47.573648787 +0000 UTC m=+0.745583476 container cleanup 5525a61ec5f77b409801ae63ce4854039259658ee4afc6eae99b2cb373607dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-06T02:31:47.738 INFO:journalctl@ceph.mon.c.smithi017.stdout:Apr 06 02:31:47 smithi017 bash[91034]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-c 2024-04-06T02:31:47.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.c.service' 2024-04-06T02:31:47.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:31:47.973 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-06T02:31:47.974 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-06T02:31:47.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.b 2024-04-06T02:31:48.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 systemd[1]: Stopping Ceph mon.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:48.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b[58664]: 2024-04-06T02:31:48.319+0000 7f5ba27c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:31:48.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b[58664]: 2024-04-06T02:31:48.319+0000 7f5ba27c7700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-06T02:31:48.659 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 podman[77197]: 2024-04-06 02:31:48.463866438 +0000 UTC m=+0.266344544 container died bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:31:48.986 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:48 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: ::ffff:172.21.15.104 - - [06/Apr/2024:02:31:48] "GET /metrics HTTP/1.1" 200 38702 "" "Prometheus/2.33.4" 2024-04-06T02:31:49.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 podman[77197]: 2024-04-06 02:31:48.970937699 +0000 UTC m=+0.773415795 container cleanup bfeac07a24d9f1dce49f20bacba7f3fa47126eb1d135c1e735cbda574018a66c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:31:49.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 06 02:31:48 smithi104 bash[77197]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mon-b 2024-04-06T02:31:49.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mon.b.service' 2024-04-06T02:31:49.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:31:49.365 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-06T02:31:49.366 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-06T02:31:49.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y 2024-04-06T02:31:49.651 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:49 smithi017 systemd[1]: Stopping Ceph mgr.y for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:49.960 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:49 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:31:49.668+0000 7fca30fa9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:31:49.960 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:49 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y[60257]: 2024-04-06T02:31:49.668+0000 7fca30fa9700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-06T02:31:49.960 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:49 smithi017 podman[91156]: 2024-04-06 02:31:49.700469044 +0000 UTC m=+0.119491509 container died fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-06T02:31:50.235 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:50 smithi017 podman[91156]: 2024-04-06 02:31:50.161587689 +0000 UTC m=+0.580610157 container cleanup fc819800a3730bb42acbf976593d04953bb4aec882e727923ee0c3e6e07890d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-06T02:31:50.236 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Apr 06 02:31:50 smithi017 bash[91156]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-y 2024-04-06T02:31:50.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.y.service' 2024-04-06T02:31:50.612 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:31:50.613 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-06T02:31:50.613 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-06T02:31:50.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x 2024-04-06T02:31:50.909 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:50 smithi104 systemd[1]: Stopping Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:51.274 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:50 smithi104 podman[77319]: 2024-04-06 02:31:50.956722879 +0000 UTC m=+0.129318500 container died 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, org.label-schema.license=GPLv2) 2024-04-06T02:31:51.556 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 podman[77319]: 2024-04-06 02:31:51.384097254 +0000 UTC m=+0.556692878 container cleanup 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-06T02:31:51.556 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 bash[77319]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x 2024-04-06T02:31:51.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service' 2024-04-06T02:31:51.810 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 podman[77333]: 2024-04-06 02:31:51.57534039 +0000 UTC m=+0.617268303 container remove 9b973d332e81435203bf4b323e120d74b6a2e9a033406ddf9bc07e3a84df67da (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-mgr-x, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:31:51.810 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:31:51.810 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Failed with result 'exit-code'. 2024-04-06T02:31:51.810 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 systemd[1]: Stopped Ceph mgr.x for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:31:51.810 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Apr 06 02:31:51 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@mgr.x.service: Consumed 20.025s CPU time. 2024-04-06T02:31:51.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:31:51.823 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-06T02:31:51.823 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-06T02:31:51.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.0 2024-04-06T02:31:52.157 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:51 smithi017 systemd[1]: Stopping Ceph osd.0 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:31:52.485 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:52 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:31:52.174+0000 7fb0aa521700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:31:52.486 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:52 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:31:52.174+0000 7fb0aa521700 -1 osd.0 158 *** Got signal Terminated *** 2024-04-06T02:31:52.486 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:52 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0[70359]: 2024-04-06T02:31:52.174+0000 7fb0aa521700 -1 osd.0 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:31:57.975 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:57 smithi017 podman[91276]: 2024-04-06 02:31:57.675356245 +0000 UTC m=+5.638556475 container died d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-06T02:31:58.236 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:58 smithi017 podman[91276]: 2024-04-06 02:31:58.083902107 +0000 UTC m=+6.047102338 container cleanup d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-06T02:31:58.236 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:58 smithi017 bash[91276]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T02:31:58.735 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:58 smithi017 podman[91293]: 2024-04-06 02:31:58.452056242 +0000 UTC m=+0.770745257 container remove d8143254bdeb20a39dcc542a22555c1784d730483026c1ad4d6aae80f72ce26c (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-9f19235, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-06T02:31:59.103 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:58 smithi017 podman[91387]: 2024-04-06 02:31:58.761750789 +0000 UTC m=+0.017297417 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:31:59.104 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:58 smithi017 podman[91387]: 2024-04-06 02:31:58.894834723 +0000 UTC m=+0.150381373 container create d5fbeeab342880f385c47b3246c1b1f0f5764938037315c76651b850d16dc571 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:31:59.104 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:59 smithi017 podman[91387]: 2024-04-06 02:31:59.122387462 +0000 UTC m=+0.377934073 container init d5fbeeab342880f385c47b3246c1b1f0f5764938037315c76651b850d16dc571 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-06T02:31:59.405 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:59 smithi017 podman[91387]: 2024-04-06 02:31:59.127109685 +0000 UTC m=+0.382656293 container start d5fbeeab342880f385c47b3246c1b1f0f5764938037315c76651b850d16dc571 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:31:59.405 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:59 smithi017 podman[91387]: 2024-04-06 02:31:59.214245744 +0000 UTC m=+0.469792369 container attach d5fbeeab342880f385c47b3246c1b1f0f5764938037315c76651b850d16dc571 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-06T02:31:59.724 INFO:journalctl@ceph.osd.0.smithi017.stdout:Apr 06 02:31:59 smithi017 podman[91414]: 2024-04-06 02:31:59.454964681 +0000 UTC m=+0.021923513 container died d5fbeeab342880f385c47b3246c1b1f0f5764938037315c76651b850d16dc571 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-06T02:32:00.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-06T02:32:00.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:00.101 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-06T02:32:00.101 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-06T02:32:00.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.1 2024-04-06T02:32:00.334 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:00 smithi017 systemd[1]: Stopping Ceph osd.1 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:00.736 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:32:00.462+0000 7fb25ec0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:00.736 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:32:00.462+0000 7fb25ec0e700 -1 osd.1 158 *** Got signal Terminated *** 2024-04-06T02:32:00.736 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:00 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1[72813]: 2024-04-06T02:32:00.462+0000 7fb25ec0e700 -1 osd.1 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:04.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:04.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:04.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:04 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:04.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:06.103 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:05 smithi017 podman[91506]: 2024-04-06 02:32:05.823808249 +0000 UTC m=+5.473574625 container died b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:32:06.404 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:06 smithi017 podman[91506]: 2024-04-06 02:32:06.230287667 +0000 UTC m=+5.880054027 container cleanup b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-06T02:32:06.404 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:06 smithi017 bash[91506]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T02:32:06.735 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:06 smithi017 podman[91523]: 2024-04-06 02:32:06.423215754 +0000 UTC m=+0.597609343 container remove b86017a8507a9724bf1efe9ed582a4be6b865524ab977c0a354d46e1f5fc85be (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-06T02:32:07.235 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:06 smithi017 podman[91619]: 2024-04-06 02:32:06.760007758 +0000 UTC m=+0.018412046 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:07.236 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:06 smithi017 podman[91619]: 2024-04-06 02:32:06.865752712 +0000 UTC m=+0.124157001 container create 978c4b417162169edc62a09edc36bbebbc83f50c368ae2799c8fe644d246605e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-06T02:32:07.236 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:07 smithi017 podman[91619]: 2024-04-06 02:32:07.026406391 +0000 UTC m=+0.284810679 container init 978c4b417162169edc62a09edc36bbebbc83f50c368ae2799c8fe644d246605e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:32:07.236 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:07 smithi017 podman[91619]: 2024-04-06 02:32:07.029540568 +0000 UTC m=+0.287944860 container start 978c4b417162169edc62a09edc36bbebbc83f50c368ae2799c8fe644d246605e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-06T02:32:07.236 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:07 smithi017 podman[91619]: 2024-04-06 02:32:07.084733187 +0000 UTC m=+0.343137504 container attach 978c4b417162169edc62a09edc36bbebbc83f50c368ae2799c8fe644d246605e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-9f19235, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:32:07.594 INFO:journalctl@ceph.osd.1.smithi017.stdout:Apr 06 02:32:07 smithi017 podman[91619]: 2024-04-06 02:32:07.32847846 +0000 UTC m=+0.586882740 container died 978c4b417162169edc62a09edc36bbebbc83f50c368ae2799c8fe644d246605e (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.schema-version=1.0) 2024-04-06T02:32:07.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-06T02:32:07.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:07.940 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-06T02:32:07.940 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-06T02:32:07.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.2 2024-04-06T02:32:08.235 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:08 smithi017 systemd[1]: Stopping Ceph osd.2 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:08.736 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:32:08.291+0000 7fb2b2d80700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:08.736 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:32:08.291+0000 7fb2b2d80700 -1 osd.2 158 *** Got signal Terminated *** 2024-04-06T02:32:08.736 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:08 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2[75135]: 2024-04-06T02:32:08.291+0000 7fb2b2d80700 -1 osd.2 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:14.040 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:13 smithi017 podman[91725]: 2024-04-06 02:32:13.776387208 +0000 UTC m=+5.587504857 container died 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-06T02:32:14.365 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91725]: 2024-04-06 02:32:14.145852855 +0000 UTC m=+5.956970544 container cleanup 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-06T02:32:14.365 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 bash[91725]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T02:32:14.366 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=error ts=2024-04-06T02:32:14.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:14.366 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:14.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:14.366 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:14 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:14.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:14.692 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91742]: 2024-04-06 02:32:14.384361676 +0000 UTC m=+0.606433396 container remove 0335ae11636fa7d599fc35238190e1fe82bc9ff44d4f68ac7de847369f66dda0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, org.label-schema.build-date=20240403) 2024-04-06T02:32:14.985 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91835]: 2024-04-06 02:32:14.614329404 +0000 UTC m=+0.018110670 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:14.986 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91835]: 2024-04-06 02:32:14.727155632 +0000 UTC m=+0.130936899 container create bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-06T02:32:14.986 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91835]: 2024-04-06 02:32:14.946527238 +0000 UTC m=+0.350308505 container init bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-06T02:32:14.986 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:14 smithi017 podman[91835]: 2024-04-06 02:32:14.949890287 +0000 UTC m=+0.353671554 container start bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-9f19235, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-06T02:32:15.485 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:15 smithi017 podman[91835]: 2024-04-06 02:32:15.029795874 +0000 UTC m=+0.433577155 container attach bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-06T02:32:15.486 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:15 smithi017 podman[91835]: 2024-04-06 02:32:15.246538102 +0000 UTC m=+0.650319378 container died bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-06T02:32:15.873 INFO:journalctl@ceph.osd.2.smithi017.stdout:Apr 06 02:32:15 smithi017 podman[91861]: 2024-04-06 02:32:15.866837977 +0000 UTC m=+0.615483859 container remove bf75bc9c71eaaeb8e1430cc9ed8614b14d799ba433767ed46a2a667ef420adf3 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-9f19235, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-06T02:32:15.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-06T02:32:15.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:15.920 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-06T02:32:15.920 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-06T02:32:15.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.3 2024-04-06T02:32:16.234 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:16 smithi017 systemd[1]: Stopping Ceph osd.3 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:16.485 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:32:16.252+0000 7f5da34c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:16.485 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:32:16.252+0000 7f5da34c5700 -1 osd.3 158 *** Got signal Terminated *** 2024-04-06T02:32:16.485 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:16 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:32:16.252+0000 7f5da34c5700 -1 osd.3 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:18.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:18 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:18.314+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:19.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:19.361+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:20.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:19 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:19.905+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:20.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:20.393+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:20.985 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:20 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3[77597]: 2024-04-06T02:32:20.693+0000 7f5d9bab1700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.979883+0000 front 2024-04-06T02:31:56.979924+0000 (oldest deadline 2024-04-06T02:32:19.880056+0000) 2024-04-06T02:32:21.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:20 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:20.864+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:21.425 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:21.354+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:21.846 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:21.443+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:21.960 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:21 smithi017 podman[91940]: 2024-04-06 02:32:21.717620592 +0000 UTC m=+5.545356005 container died 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-06T02:32:22.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:21 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:21.864+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:22.210 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[91940]: 2024-04-06 02:32:22.052847438 +0000 UTC m=+5.880582851 container cleanup 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-06T02:32:22.210 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 bash[91940]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T02:32:22.485 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[91957]: 2024-04-06 02:32:22.229170355 +0000 UTC m=+0.507951112 container remove 85c4daa4b795f377410d605d034e9ff6a649f9ba240aab691d32a2447ea1f478 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9f19235, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-06T02:32:22.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:22.326+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:22.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:22.471+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:22.930 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[92049]: 2024-04-06 02:32:22.55331336 +0000 UTC m=+0.018126674 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:22.931 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[92049]: 2024-04-06 02:32:22.654934736 +0000 UTC m=+0.119748042 container create 12d86802378de0b6118fbed65b58b1082d3cb04ad7009f499fab05155029c4f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-06T02:32:22.931 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[92049]: 2024-04-06 02:32:22.865768803 +0000 UTC m=+0.330582112 container init 12d86802378de0b6118fbed65b58b1082d3cb04ad7009f499fab05155029c4f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-9f19235, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-06T02:32:22.931 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[92049]: 2024-04-06 02:32:22.869019147 +0000 UTC m=+0.333832456 container start 12d86802378de0b6118fbed65b58b1082d3cb04ad7009f499fab05155029c4f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-06T02:32:22.931 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:22 smithi017 podman[92049]: 2024-04-06 02:32:22.949102505 +0000 UTC m=+0.413915814 container attach 12d86802378de0b6118fbed65b58b1082d3cb04ad7009f499fab05155029c4f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-06T02:32:23.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:22.869+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:23.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:22 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:22.785+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:23.235 INFO:journalctl@ceph.osd.3.smithi017.stdout:Apr 06 02:32:23 smithi017 podman[92049]: 2024-04-06 02:32:23.179077085 +0000 UTC m=+0.643890384 container died 12d86802378de0b6118fbed65b58b1082d3cb04ad7009f499fab05155029c4f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-06T02:32:23.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:23.366+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:23.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:23.440+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:23.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-06T02:32:23.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:23.813 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-06T02:32:23.814 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-06T02:32:23.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.4 2024-04-06T02:32:24.121 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:23.784+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:24.121 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:23 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:23.852+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:24.121 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:23 smithi104 systemd[1]: Stopping Ceph osd.4 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:24.409 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:24.356+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:24.409 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:24.139+0000 7f73800ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:24.409 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:24.139+0000 7f73800ed700 -1 osd.4 158 *** Got signal Terminated *** 2024-04-06T02:32:24.409 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:24.139+0000 7f73800ed700 -1 osd.4 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:24.735 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=error ts=2024-04-06T02:32:24.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:24.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:24.266Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:24.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:24 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:24.267Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:24.747 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:24.437+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:25.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:24.849+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:25.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:24 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:24.766+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:25.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:25.326+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:25.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:25.437+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:26.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:25.849+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:26.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:25 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:25.816+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:26.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:26 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:26.463+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:26.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:26 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:26.361+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:27.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:26 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:26.839+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:27.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:26 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:26.784+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:27.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:27.451+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:27.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:27.331+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:28.160 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:27.817+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:28.160 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:27 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:27.862+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:28.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:28.380+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:28.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:28.451+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:28.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:28.451+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:29.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4[62295]: 2024-04-06T02:32:28.853+0000 7f737c6e1700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:53.876849+0000 front 2024-04-06T02:31:53.876636+0000 (oldest deadline 2024-04-06T02:32:19.776809+0000) 2024-04-06T02:32:29.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:28.767+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:29.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:28 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:28.767+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:29.659 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:29 smithi104 podman[77458]: 2024-04-06 02:32:29.524562241 +0000 UTC m=+5.497982866 container died 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-06T02:32:29.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:29.402+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:29.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:29.402+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:29.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:29.419+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:29.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:29.419+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:30.146 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:29 smithi104 podman[77458]: 2024-04-06 02:32:29.947483647 +0000 UTC m=+5.920904271 container cleanup 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:32:30.147 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:29 smithi104 bash[77458]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T02:32:30.147 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:29.760+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:30.147 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:29 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:29.760+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:30.409 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:30 smithi104 podman[77475]: 2024-04-06 02:32:30.16558892 +0000 UTC m=+0.638748123 container remove 6cfd45ba23db6e53e100529fd5ed38c4307df9ce39fc2390183e8d72784b0d8b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235) 2024-04-06T02:32:30.409 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:30.360+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:30.409 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:30.360+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:30.409 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:30.378+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:30.409 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:30.378+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:30.846 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:30 smithi104 podman[77566]: 2024-04-06 02:32:30.469337973 +0000 UTC m=+0.017924594 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:30.846 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:30 smithi104 podman[77566]: 2024-04-06 02:32:30.758614986 +0000 UTC m=+0.307201649 container create 8f3a1d29392c5e001ecdcbdacbee1a3d74d4cb0289beea46a753cfefe0f9959a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-06T02:32:30.847 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:30.727+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:30.847 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:30 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:30.727+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:31.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:30 smithi104 podman[77566]: 2024-04-06 02:32:30.961109935 +0000 UTC m=+0.509696556 container init 8f3a1d29392c5e001ecdcbdacbee1a3d74d4cb0289beea46a753cfefe0f9959a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T02:32:31.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:30 smithi104 podman[77566]: 2024-04-06 02:32:30.964275734 +0000 UTC m=+0.512862356 container start 8f3a1d29392c5e001ecdcbdacbee1a3d74d4cb0289beea46a753cfefe0f9959a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-06T02:32:31.159 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:31 smithi104 podman[77566]: 2024-04-06 02:32:31.044174082 +0000 UTC m=+0.592760708 container attach 8f3a1d29392c5e001ecdcbdacbee1a3d74d4cb0289beea46a753cfefe0f9959a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-06T02:32:31.571 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 06 02:32:31 smithi104 podman[77566]: 2024-04-06 02:32:31.260776569 +0000 UTC m=+0.809363189 container died 8f3a1d29392c5e001ecdcbdacbee1a3d74d4cb0289beea46a753cfefe0f9959a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-06T02:32:31.571 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:31.364+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:31.571 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:31.364+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:31.571 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:31.356+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:31.572 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:31.356+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:31.887 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:31.768+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:31.887 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:31 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:31.768+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:31.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-06T02:32:31.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:31.952 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-06T02:32:31.952 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-06T02:32:31.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.5 2024-04-06T02:32:32.159 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 systemd[1]: Stopping Ceph osd.5 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:32.159 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:32.142Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-04-06T02:32:32.159 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:32.149Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-04-06T02:32:32.159 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:32.149Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-04-06T02:32:32.159 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:32.149Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-04-06T02:32:32.159 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:32.149Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-04-06T02:32:32.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:32.346+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:32.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:32.347+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:32.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:32.308+0000 7fbfa6832700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:32.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:32.308+0000 7fbfa6832700 -1 osd.5 158 *** Got signal Terminated *** 2024-04-06T02:32:32.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:32.308+0000 7fbfa6832700 -1 osd.5 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:32.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:32.405+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:32.660 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:32.405+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:33.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:32.735+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:33.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:32 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:32.735+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:33.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:33.413+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:33.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:33.413+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:33.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:33.382+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:33.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:33.382+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:34.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:33.732+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:34.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:33 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:33.732+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:34.647 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:34 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=error ts=2024-04-06T02:32:34.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:34.648 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:34 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:34.266Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:34.648 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:34 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:34.267Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:34.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:34.446+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:34.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:34.446+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:34.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:34.357+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:34.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:34.357+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:35.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:34.699+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:35.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:34.699+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:35.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:34 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:34.699+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:35.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:35.462+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:35.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:35.462+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:35.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:35.462+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:13.182289+0000 front 2024-04-06T02:32:13.182242+0000 (oldest deadline 2024-04-06T02:32:34.882069+0000) 2024-04-06T02:32:35.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:35.333+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:35.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:35.333+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:36.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:35.721+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:36.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:35.721+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:36.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:35 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:35.721+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:36.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:36.482+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.880337+0000 front 2024-04-06T02:31:56.880207+0000 (oldest deadline 2024-04-06T02:32:20.980185+0000) 2024-04-06T02:32:36.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:36.482+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.880856+0000 front 2024-04-06T02:32:03.880962+0000 (oldest deadline 2024-04-06T02:32:28.580923+0000) 2024-04-06T02:32:36.659 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5[64573]: 2024-04-06T02:32:36.482+0000 7fbfa2e26700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:13.182289+0000 front 2024-04-06T02:32:13.182242+0000 (oldest deadline 2024-04-06T02:32:34.882069+0000) 2024-04-06T02:32:36.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:36.324+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:36.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:36.324+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:36.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:36.324+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:37.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:36.730+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:37.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:36.730+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:37.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:36 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:36.730+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:37.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:37.358+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:37.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:37.358+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:37.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:37.358+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:38.057 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:37 smithi104 podman[77682]: 2024-04-06 02:32:37.83035712 +0000 UTC m=+5.632488956 container died 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, org.label-schema.license=GPLv2) 2024-04-06T02:32:38.057 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:37.681+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:38.058 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:37.681+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:38.058 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:37 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:37.681+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:38.386 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:38 smithi104 podman[77682]: 2024-04-06 02:32:38.183689933 +0000 UTC m=+5.985821773 container cleanup 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, RELEASE=squid-9f19235, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-06T02:32:38.387 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:38 smithi104 bash[77682]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T02:32:38.642 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:38 smithi104 podman[77700]: 2024-04-06 02:32:38.459914005 +0000 UTC m=+0.631766554 container remove 7e9ae60791865b5086a1bf0af19bd2669f0ad715a282cb81ad10967b9d59690a (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-06T02:32:38.642 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:38.404+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:38.642 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:38.404+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:38.642 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:38.404+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:38.902 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:38 smithi104 podman[77789]: 2024-04-06 02:32:38.709547989 +0000 UTC m=+0.018291921 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:38.902 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:38 smithi104 podman[77789]: 2024-04-06 02:32:38.818984653 +0000 UTC m=+0.127728567 container create 10d8b7f515a4b2a659380ba87515efb4136e4f39abb9f3adb06dd6b2df5da237 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-9f19235, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-06T02:32:38.902 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:38.660+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:38.902 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:38.660+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:38.903 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:38 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:38.660+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:39.159 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:39 smithi104 podman[77789]: 2024-04-06 02:32:39.029700275 +0000 UTC m=+0.338444186 container init 10d8b7f515a4b2a659380ba87515efb4136e4f39abb9f3adb06dd6b2df5da237 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-06T02:32:39.159 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:39 smithi104 podman[77789]: 2024-04-06 02:32:39.032772192 +0000 UTC m=+0.341516108 container start 10d8b7f515a4b2a659380ba87515efb4136e4f39abb9f3adb06dd6b2df5da237 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-06T02:32:39.160 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:39 smithi104 podman[77789]: 2024-04-06 02:32:39.104556431 +0000 UTC m=+0.413300349 container attach 10d8b7f515a4b2a659380ba87515efb4136e4f39abb9f3adb06dd6b2df5da237 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-9f19235) 2024-04-06T02:32:39.589 INFO:journalctl@ceph.osd.5.smithi104.stdout:Apr 06 02:32:39 smithi104 podman[77814]: 2024-04-06 02:32:39.349082914 +0000 UTC m=+0.017993483 container died 10d8b7f515a4b2a659380ba87515efb4136e4f39abb9f3adb06dd6b2df5da237 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403) 2024-04-06T02:32:39.589 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:39.450+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:39.589 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:39.450+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:39.589 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:39.450+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:39.863 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:39.677+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:39.864 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:39.677+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:39.864 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:39 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:39.677+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:39.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-06T02:32:39.941 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:39.941 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-06T02:32:39.941 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-06T02:32:39.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.6 2024-04-06T02:32:40.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 systemd[1]: Stopping Ceph osd.6 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:40.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.285+0000 7fe66b9f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:40.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.285+0000 7fe66b9f9700 -1 osd.6 158 *** Got signal Terminated *** 2024-04-06T02:32:40.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.285+0000 7fe66b9f9700 -1 osd.6 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:40.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:40.473+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:40.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:40.473+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:40.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:40.473+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:41.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.706+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:41.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.706+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:41.159 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:40 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:40.706+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:41.814 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:41.461+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:41.814 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:41.461+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:41.814 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:41.461+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:41.815 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:41.674+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:41.815 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:41.674+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:41.815 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:41 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:41.674+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:42.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:42.646+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:42.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:42.646+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:42.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:42.646+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:42.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:42.646+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:16.237867+0000 front 2024-04-06T02:32:16.237960+0000 (oldest deadline 2024-04-06T02:32:42.137747+0000) 2024-04-06T02:32:42.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:42.464+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:42.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:42.464+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:42.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:42.464+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:42.660 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:42 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:42.464+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:43.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:43.465+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:43.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:43.465+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:43.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:43.465+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:43.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:43.465+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:43.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:43.696+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:43.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:43.696+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:43.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:43.696+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:43.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:43 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:43.696+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:16.237867+0000 front 2024-04-06T02:32:16.237960+0000 (oldest deadline 2024-04-06T02:32:42.137747+0000) 2024-04-06T02:32:44.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:44 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=error ts=2024-04-06T02:32:44.267Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:44.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:44 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:44.268Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:44.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:44 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:44.268Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:44.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:44.512+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:44.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:44.512+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:44.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:44.512+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:44.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:44.512+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:44.909 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:44.709+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:56.935627+0000 front 2024-04-06T02:31:56.935567+0000 (oldest deadline 2024-04-06T02:32:22.235383+0000) 2024-04-06T02:32:44.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:44.709+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:02.236074+0000 front 2024-04-06T02:32:02.235855+0000 (oldest deadline 2024-04-06T02:32:28.135968+0000) 2024-04-06T02:32:44.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:44.709+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:08.637242+0000 front 2024-04-06T02:32:08.637159+0000 (oldest deadline 2024-04-06T02:32:34.537047+0000) 2024-04-06T02:32:44.910 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:44 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6[66841]: 2024-04-06T02:32:44.709+0000 7fe663fe5700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:16.237867+0000 front 2024-04-06T02:32:16.237960+0000 (oldest deadline 2024-04-06T02:32:42.137747+0000) 2024-04-06T02:32:45.909 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:45 smithi104 podman[77894]: 2024-04-06 02:32:45.712677314 +0000 UTC m=+5.531544595 container died 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-06T02:32:45.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:45.507+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:45.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:45.507+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:45.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:45.508+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:45.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:45 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:45.508+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:46.401 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:46 smithi104 podman[77894]: 2024-04-06 02:32:46.185269595 +0000 UTC m=+6.004136876 container cleanup 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-06T02:32:46.401 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:46 smithi104 bash[77894]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T02:32:46.659 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:46 smithi104 podman[77911]: 2024-04-06 02:32:46.420011098 +0000 UTC m=+0.696411699 container remove 8eda2c82cb18e9c5c37713d7148aac326c717f0e2227973e6e2a93a9f526aa74 (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9f19235, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-06T02:32:46.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:46.516+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:46.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:46.516+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:46.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:46.516+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:46.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:46 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:46.516+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:46.988 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:46 smithi104 podman[78003]: 2024-04-06 02:32:46.649332148 +0000 UTC m=+0.017707566 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:46.988 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:46 smithi104 podman[78003]: 2024-04-06 02:32:46.770955151 +0000 UTC m=+0.139330604 container create d35a4636969891f3aaa6ec8bff5f68d87a91a4e719342f8205a7a205f20e3d5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-06T02:32:46.988 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:47 smithi104 podman[78003]: 2024-04-06 02:32:47.007366973 +0000 UTC m=+0.375742390 container init d35a4636969891f3aaa6ec8bff5f68d87a91a4e719342f8205a7a205f20e3d5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5) 2024-04-06T02:32:47.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:47 smithi104 podman[78003]: 2024-04-06 02:32:47.010506059 +0000 UTC m=+0.378881476 container start d35a4636969891f3aaa6ec8bff5f68d87a91a4e719342f8205a7a205f20e3d5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-06T02:32:47.277 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:47 smithi104 podman[78003]: 2024-04-06 02:32:47.098373207 +0000 UTC m=+0.466748631 container attach d35a4636969891f3aaa6ec8bff5f68d87a91a4e719342f8205a7a205f20e3d5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9f19235, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-06T02:32:47.565 INFO:journalctl@ceph.osd.6.smithi104.stdout:Apr 06 02:32:47 smithi104 podman[78003]: 2024-04-06 02:32:47.296359927 +0000 UTC m=+0.664735336 container died d35a4636969891f3aaa6ec8bff5f68d87a91a4e719342f8205a7a205f20e3d5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-9f19235) 2024-04-06T02:32:47.566 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:47.487+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:47.566 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:47.487+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:47.566 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:47.488+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:47.566 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:47 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:47.488+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:47.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-06T02:32:47.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:47.985 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-06T02:32:47.985 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-06T02:32:47.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.7 2024-04-06T02:32:48.277 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 systemd[1]: Stopping Ceph osd.7 for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.295+0000 7fbfcc4a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.295+0000 7fbfcc4a0700 -1 osd.7 158 *** Got signal Terminated *** 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.295+0000 7fbfcc4a0700 -1 osd.7 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.501+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.501+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.501+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:48.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:48 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:48.501+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:49.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:49 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:49.460+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:49.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:49 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:49.460+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:49.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:49 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:49.460+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:49.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:49 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:49.460+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:50.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:50.456+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:50.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:50.456+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:50.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:50.456+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:50.909 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:50 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:50.456+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:51.810 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:51.444+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:51.810 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:51.444+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:51.810 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:51.444+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:51.810 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:51 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:51.444+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:52.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:52 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:52.457+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-04-06T02:31:55.435140+0000 front 2024-04-06T02:31:55.435105+0000 (oldest deadline 2024-04-06T02:32:17.735061+0000) 2024-04-06T02:32:52.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:52 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:52.457+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-04-06T02:32:03.636079+0000 front 2024-04-06T02:32:03.636140+0000 (oldest deadline 2024-04-06T02:32:28.336006+0000) 2024-04-06T02:32:52.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:52 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:52.457+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-04-06T02:32:10.637063+0000 front 2024-04-06T02:32:10.636977+0000 (oldest deadline 2024-04-06T02:32:35.936969+0000) 2024-04-06T02:32:52.659 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:52 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7[69138]: 2024-04-06T02:32:52.457+0000 7fbfc4a8c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-04-06T02:32:17.638158+0000 front 2024-04-06T02:32:17.638131+0000 (oldest deadline 2024-04-06T02:32:42.338093+0000) 2024-04-06T02:32:54.036 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:53 smithi104 podman[78109]: 2024-04-06 02:32:53.80245415 +0000 UTC m=+5.611795563 container died 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-9f19235, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-06T02:32:54.409 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:54 smithi104 podman[78109]: 2024-04-06 02:32:54.245649667 +0000 UTC m=+6.054991069 container cleanup 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-06T02:32:54.409 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:54 smithi104 bash[78109]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T02:32:54.735 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=error ts=2024-04-06T02:32:54.268Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:54.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:54.268Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-04-06T02:32:54.736 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:54 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=warn ts=2024-04-06T02:32:54.268Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-04-06T02:32:54.828 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:54 smithi104 podman[78140]: 2024-04-06 02:32:54.446900972 +0000 UTC m=+0.639753462 container remove 3972a8bf83b368d7724e149c145e5257c2090468d9dc6de848da31195b82e7ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9f19235, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-06T02:32:55.105 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:54 smithi104 podman[78234]: 2024-04-06 02:32:54.847745253 +0000 UTC m=+0.114266327 container create c5601f7e829da81a0dc42dc77f2efcff53bfae71441722512e631b27082d517d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-9f19235, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-06T02:32:55.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:54 smithi104 podman[78234]: 2024-04-06 02:32:54.751330548 +0000 UTC m=+0.017851587 image pull a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880 quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T02:32:55.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:55 smithi104 podman[78234]: 2024-04-06 02:32:55.050104546 +0000 UTC m=+0.316625577 container init c5601f7e829da81a0dc42dc77f2efcff53bfae71441722512e631b27082d517d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:32:55.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:55 smithi104 podman[78234]: 2024-04-06 02:32:55.053062342 +0000 UTC m=+0.319583370 container start c5601f7e829da81a0dc42dc77f2efcff53bfae71441722512e631b27082d517d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-9f19235, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-06T02:32:55.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:55 smithi104 podman[78234]: 2024-04-06 02:32:55.124779444 +0000 UTC m=+0.391300484 container attach c5601f7e829da81a0dc42dc77f2efcff53bfae71441722512e631b27082d517d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-9f19235, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-06T02:32:55.409 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 06 02:32:55 smithi104 podman[78234]: 2024-04-06 02:32:55.346676314 +0000 UTC m=+0.613197343 container died c5601f7e829da81a0dc42dc77f2efcff53bfae71441722512e631b27082d517d (image=quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-9f19235, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-06T02:32:55.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-06T02:32:55.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:55.925 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-06T02:32:55.925 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-06T02:32:55.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a 2024-04-06T02:32:56.128 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 systemd[1]: Stopping Ceph prometheus.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:56.409 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:32:56.409 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-06T02:32:56.409 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-06T02:32:56.409 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-06T02:32:56.409 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a[54171]: ts=2024-04-06T02:32:56.147Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-06T02:32:56.410 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 podman[78313]: 2024-04-06 02:32:56.157600726 +0000 UTC m=+0.124665027 container died 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:56.770 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 podman[78313]: 2024-04-06 02:32:56.555367528 +0000 UTC m=+0.522431828 container cleanup 9498923da881f37bc61de029cf14ae9aff7fa2babfdb2d5cb62630bb419955e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:56.771 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 bash[78313]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T02:32:56.771 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Apr 06 02:32:56 smithi104 bash[78340]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-prometheus.a" found: no such container 2024-04-06T02:32:56.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@prometheus.a.service' 2024-04-06T02:32:56.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T02:32:56.813 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-06T02:32:56.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T02:32:58.669 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 systemd[1]: Stopping Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:58.669 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 podman[92418]: 2024-04-06 02:32:58.448137622 +0000 UTC m=+0.105608411 container died f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:58.927 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 podman[92418]: 2024-04-06 02:32:58.778933963 +0000 UTC m=+0.436404750 container cleanup f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:58.927 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 bash[92418]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a 2024-04-06T02:32:59.235 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 systemd[1]: Stopping Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:32:59.235 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 podman[92432]: 2024-04-06 02:32:58.946788936 +0000 UTC m=+0.503062476 container remove f9dfc8b2fd49e542d2a0cbef6adee6bff4628780b9b6f4c17b401a615890a191 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:59.236 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:32:59.236 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 bash[92444]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.a" found: no such container 2024-04-06T02:32:59.236 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-06T02:32:59.236 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 systemd[1]: Stopped Ceph node-exporter.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:32:59.236 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Apr 06 02:32:58 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.a.service: Consumed 3.512s CPU time. 2024-04-06T02:32:59.573 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a[58998]: level=info ts=2024-04-06T02:32:59.291Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-06T02:32:59.574 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 podman[92497]: 2024-04-06 02:32:59.332454569 +0000 UTC m=+0.145055834 container died cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:59.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 podman[92497]: 2024-04-06 02:32:59.692344825 +0000 UTC m=+0.504946098 container cleanup cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:32:59.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 bash[92497]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T02:32:59.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 bash[92523]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager.a" found: no such container 2024-04-06T02:33:00.213 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:32:59 smithi017 podman[92511]: 2024-04-06 02:32:59.960088072 +0000 UTC m=+0.620147826 container remove cbef90f04e7862f0b3d7409f0a39c4b3e9dbed30f97c8b8da9e7863f12d3e8cb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-06T02:33:00.213 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:33:00 smithi017 podman[92511]: 2024-04-06 02:33:00.035721283 +0000 UTC m=+0.695781037 volume remove e448428390426f70e129cee2b74c89b9d3b8c571108d28d0a40dc9fa9eb00e2c 2024-04-06T02:33:00.213 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:33:00 smithi017 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-06T02:33:00.213 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Apr 06 02:33:00 smithi017 systemd[1]: Stopped Ceph alertmanager.a for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:33:34.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T02:33:36.284 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 systemd[1]: Stopping Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4... 2024-04-06T02:33:36.284 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 podman[78673]: 2024-04-06 02:33:36.130312334 +0000 UTC m=+0.115715244 container died e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:33:36.659 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 podman[78673]: 2024-04-06 02:33:36.510931495 +0000 UTC m=+0.496334394 container cleanup e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:33:36.659 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 bash[78673]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b 2024-04-06T02:33:36.659 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 bash[78699]: Error: no container with name or ID "ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter.b" found: no such container 2024-04-06T02:33:36.962 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 podman[78687]: 2024-04-06 02:33:36.787309527 +0000 UTC m=+0.657411232 container remove e14e1595bb239ae0b1bb8442bd40d2178ea1330244ecc9dcf18de23831dea47a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-06T02:33:36.962 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-06T02:33:36.962 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-06T02:33:36.962 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 systemd[1]: Stopped Ceph node-exporter.b for dbfad14a-f3ba-11ee-b649-cb9ed24678a4. 2024-04-06T02:33:36.962 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Apr 06 02:33:36 smithi104 systemd[1]: ceph-dbfad14a-f3ba-11ee-b649-cb9ed24678a4@node-exporter.b.service: Consumed 3.674s CPU time. 2024-04-06T02:34:00.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:34:00.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:34:00.088 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-06T02:34:00.090 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi017/crash 2024-04-06T02:34:00.091 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T02:34:00.149 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T02:34:00.149 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2024-04-06T02:34:00.150 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi104/crash 2024-04-06T02:34:00.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T02:34:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T02:34:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-04-06T02:34:00.192 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-06T02:34:00.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T02:34:00.222 INFO:teuthology.orchestra.run.smithi017.stdout:2024-04-06T02:12:50.871332+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-06T02:34:00.222 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-06T02:34:00.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[SEC\]' /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T02:34:00.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]' /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T02:34:00.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[WRN\]' /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-06T02:34:00.429 INFO:teuthology.orchestra.run.smithi017.stdout:2024-04-06T02:12:50.871332+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-06T02:34:00.429 INFO:tasks.cephadm:Compressing logs... 2024-04-06T02:34:00.430 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T02:34:00.472 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T02:34:00.496 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T02:34:00.496 INFO:teuthology.orchestra.run.smithi017.stderr:: No such file or directory 2024-04-06T02:34:00.496 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.a.log 2024-04-06T02:34:00.496 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T02:34:00.499 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-06T02:34:00.499 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-06T02:34:00.499 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.y.log 2024-04-06T02:34:00.499 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/tcmu-runner.log 2024-04-06T02:34:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T02:34:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-04-06T02:34:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.6.log 2024-04-06T02:34:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi104.iurqzi.log 2024-04-06T02:34:00.501 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T02:34:00.501 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi104.iurqzi.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-06T02:34:00.503 INFO:teuthology.orchestra.run.smithi104.stderr: 75.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi104.iurqzi.log.gz 2024-04-06T02:34:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.7.log 2024-04-06T02:34:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log: /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-06T02:34:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.x.log 2024-04-06T02:34:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T02:34:00.504 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.y.log: 94.3% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T02:34:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T02:34:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T02:34:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T02:34:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log: 90.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T02:34:00.505 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T02:34:00.507 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.b.log 2024-04-06T02:34:00.507 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi104.smkgdi.log 2024-04-06T02:34:00.508 INFO:teuthology.orchestra.run.smithi017.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T02:34:00.509 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-06T02:34:00.511 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/tcmu-runner.log: --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T02:34:00.511 INFO:teuthology.orchestra.run.smithi017.stderr: 84.7% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/tcmu-runner.log.gz 2024-04-06T02:34:00.511 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T02:34:00.511 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 91.8% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T02:34:00.512 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-06T02:34:00.513 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T02:34:00.516 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.2.log: 93.0% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T02:34:00.517 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi017.ffhtpi.log 2024-04-06T02:34:00.517 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.c.log 2024-04-06T02:34:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T02:34:00.518 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi017.ffhtpi.log: 76.5% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.smithi017.ffhtpi.log.gz 2024-04-06T02:34:00.519 INFO:teuthology.orchestra.run.smithi104.stderr: 91.4% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T02:34:00.523 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi104.smkgdi.log: 76.0% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi104.smkgdi.log.gz 2024-04-06T02:34:00.530 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi017.qqrgsv.log 2024-04-06T02:34:00.537 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.c.log: /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi017.qqrgsv.log: 91.4% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T02:34:00.537 INFO:teuthology.orchestra.run.smithi017.stderr: 75.8% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-client.rgw.smpl.smithi017.qqrgsv.log.gz 2024-04-06T02:34:00.551 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-06T02:34:00.852 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-06T02:34:00.941 INFO:teuthology.orchestra.run.smithi104.stderr: 92.4% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-06T02:34:00.978 INFO:teuthology.orchestra.run.smithi017.stderr: 92.2% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-06T02:34:01.438 INFO:teuthology.orchestra.run.smithi017.stderr: 91.0% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-06T02:34:01.487 INFO:teuthology.orchestra.run.smithi017.stderr: 93.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-06T02:34:01.664 INFO:teuthology.orchestra.run.smithi104.stderr: 93.5% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-06T02:34:01.904 INFO:teuthology.orchestra.run.smithi104.stderr: 94.0% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-06T02:34:01.918 INFO:teuthology.orchestra.run.smithi104.stderr: 93.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-06T02:34:02.154 INFO:teuthology.orchestra.run.smithi104.stderr: 93.7% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-06T02:34:02.156 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-06T02:34:02.156 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m1.670s 2024-04-06T02:34:02.156 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m5.949s 2024-04-06T02:34:02.156 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.237s 2024-04-06T02:34:02.162 INFO:teuthology.orchestra.run.smithi017.stderr: 93.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-06T02:34:02.401 INFO:teuthology.orchestra.run.smithi017.stderr: 93.6% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-06T02:34:02.512 INFO:teuthology.orchestra.run.smithi017.stderr: 93.7% -- replaced with /var/log/ceph/dbfad14a-f3ba-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-06T02:34:02.514 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-04-06T02:34:02.514 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m2.031s 2024-04-06T02:34:02.514 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m8.043s 2024-04-06T02:34:02.514 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.293s 2024-04-06T02:34:02.514 INFO:tasks.cephadm:Archiving logs... 2024-04-06T02:34:02.515 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi017/log 2024-04-06T02:34:02.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T02:34:03.572 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi104/log 2024-04-06T02:34:03.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T02:34:04.446 INFO:tasks.cephadm:Removing cluster... 2024-04-06T02:34:04.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 --force 2024-04-06T02:34:04.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbfad14a-f3ba-11ee-b649-cb9ed24678a4 --force 2024-04-06T02:34:05.049 INFO:tasks.cephadm:Removing cephadm ... 2024-04-06T02:34:05.049 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T02:34:05.064 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T02:34:05.078 INFO:tasks.cephadm:Teardown complete 2024-04-06T02:34:05.079 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T02:34:05.088 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T02:34:05.088 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:34:05.107 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:34:05.118 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -279us[ -278us] +/- 83ms 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 7 377 41 -40us[ -38us] +/- 54ms 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +270us[ +271us] +/- 72ms 2024-04-06T02:34:05.123 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:34:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -249us[ -224us] +/- 83ms 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -49us[ -25us] +/- 54ms 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +287us[ +311us] +/- 72ms 2024-04-06T02:34:05.136 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:34:05.137 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T02:34:05.148 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T02:34:05.149 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T02:34:05.163 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T02:34:05.247 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T02:34:05.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T02:34:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:34:05.357 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T02:34:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:34:05.388 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T02:34:05.401 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T02:34:05.419 INFO:teuthology.task.internal:Duration was 1670.632087 seconds 2024-04-06T02:34:05.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T02:34:05.439 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T02:34:05.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T02:34:05.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T02:34:05.482 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:34:05.484 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:34:05.774 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T02:34:05.775 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-04-06T02:34:05.775 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T02:34:05.834 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-04-06T02:34:05.834 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T02:34:05.859 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T02:34:05.859 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T02:34:05.877 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T02:34:05.937 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T02:34:05.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T02:34:05.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T02:34:06.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T02:34:06.375 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T02:34:06.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T02:34:06.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T02:34:06.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T02:34:06.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T02:34:06.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T02:34:06.468 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-04-06T02:34:06.495 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-04-06T02:34:06.522 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T02:34:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:34:06.545 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T02:34:06.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:34:06.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T02:34:06.589 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T02:34:06.590 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi017 2024-04-06T02:34:06.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T02:34:06.641 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836/remote/smithi104 2024-04-06T02:34:06.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T02:34:06.687 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T02:34:06.688 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T02:34:06.691 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T02:34:06.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T02:34:06.749 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T02:34:06.749 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T02:34:06.764 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T02:34:06.765 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T02:34:06.767 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T02:34:06.779 INFO:teuthology.orchestra.run.smithi017.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 02:34 /home/ubuntu/cephtest 2024-04-06T02:34:06.789 INFO:teuthology.orchestra.run.smithi104.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 02:34 /home/ubuntu/cephtest 2024-04-06T02:34:06.791 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T02:34:06.803 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T02:34:06.834 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T02:34:06.863 DEBUG:teuthology.nuke:shortname: smithi017 2024-04-06T02:34:06.863 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T02:34:06.886 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 01:57:11.590167', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOK1pZtk6p3UHPBrlGCf0FkELt6+n1I+nY2qQNy86caQnb3ObNn5cXSqgvMLCLhK03JKTYZz8/hZNl+sEDU26cI='} 2024-04-06T02:34:06.890 DEBUG:teuthology.nuke:shortname: smithi104 2024-04-06T02:34:06.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T02:34:06.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 01:57:11.589018', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3Iwr0j9jUZp0aTpe7Fleo1rDVSdsbloWkXUPswxzGAwq/g1Sp7fiQARvCUKRc77sjbFiaH5F7i/JuiF2EQyRk='} 2024-04-06T02:34:06.943 INFO:teuthology.orchestra.console.smithi017:Power off 2024-04-06T02:34:06.944 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T02:34:06.968 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2024-04-06T02:34:06.968 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:06.983 INFO:teuthology.orchestra.console.smithi104:Power off 2024-04-06T02:34:06.984 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T02:34:06.994 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-04-06T02:34:07.005 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-04-06T02:34:07.006 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:07.028 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-06T02:34:10.995 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:11.030 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:11.106 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-04-06T02:34:11.140 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-06T02:34:15.112 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:15.141 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:15.221 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2024-04-06T02:34:15.251 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-06T02:34:15.323 INFO:teuthology.orchestra.console.smithi017:Power off completed 2024-04-06T02:34:19.254 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T02:34:19.364 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-04-06T02:34:19.465 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-04-06T02:34:19.569 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1670.6320867538452 failure_reason: '"2024-04-06T02:12:50.871332+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-06T02:34:19.569 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T02:34:19.651 INFO:teuthology.run:FAIL